Integrate with Scrapy

Scrapy is a very powerful spider framework. If you don't know about Scrapy, please refer to official website introduction. After integration with Crawlab, you can see the data captured by your spider task on the interface.

Crawlab SDK provides some 'helper' methods to better integrate your spider into Crawlab, such as saving the result data into Crawlab and so on.

Integrate with Scrapy

Find 'ITEM_PIPELINES'(variable of type 'dict') in 'settings.py', and add the following content.

ITEM_PIPELINES = {
    'crawlab.pipelines.CrawlabMongoPipeline': 888,
}

Then, start the Scrapy spider. After running, you should be able to see the result in task details - result .

© 2020 Crawlab, Made by Crawlab-Team all right reserved,powered by Gitbook该文件最后修改时间: 2020-05-21 10:30:12

results matching ""

    No results matching ""

    results matching ""

      No results matching ""