scrapy.linkextractor.LinkExtractor.extract_links - python examples

Here are the examples of the python api scrapy.linkextractor.LinkExtractor.extract_links taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

1 Examples 7

3 View Complete Implementation : home2.py
Copyright MIT License
Author : PacktPublishing
    def parse_page(self, response):
        links = LinkExtractor(canonicalize=True, unique=True).extract_links(response)
        for link in links:
            is_allowed = False
            for allowed_domain in self.allowed_domains:
                if allowed_domain in link.url:
                    is_allowed = True
            if is_allowed:
                print (link.url)