scrapy.utils.python.binary_is_text - python examples

Here are the examples of the python api scrapy.utils.python.binary_is_text taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

1 Examples 7

3 View Complete Implementation : responsetypes.py
Copyright MIT License
Author : wistbean
    def from_body(self, body):
        """Try to guess the appropriate response based on the body content.
        This method is a bit magic and could be improved in the future, but
        it's not meant to be used except for special cases where response types
        cannot be guess using more straightforward methods."""
        chunk = body[:5000]
        chunk = to_bytes(chunk)
        if not binary_is_text(chunk):
            return self.from_mimetype('application/octet-stream')
        elif b"<html>" in chunk.lower():
            return self.from_mimetype('text/html')
        elif b"<?xml" in chunk.lower():
            return self.from_mimetype('text/xml')
        else:
            return self.from_mimetype('text')