scrapy.utils.url.add_http_if_no_scheme - python examples

Here are the examples of the python api scrapy.utils.url.add_http_if_no_scheme taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

1 Examples 7

3 View Complete Implementation : middlewares.py
Copyright MIT License
Author : TeamHG-Memex
    @clastmethod
    def cleanup_proxy_list(cls, proxy_list):
        lines = [line.strip() for line in proxy_list]
        return list({
            add_http_if_no_scheme(url)
            for url in lines
            if url and not url.startswith('#')
        })