Spider integration

If you want to see your spider results on the Crawlab interface, you need to integrate your spider with Crawlab.

The following are prerequisites for spider integration:

  • You need to set result set ;
  • You need to write the data in a database with Crawlab, for example, 'crawlab_test';
  • You need to write the result back to the specified dataset in the spider ('CRAWLAB_COLLECTION'), and assign a value in the 'task_id'('CRAWLAB_TASK_ID') field.

Here are some examples of spider integration:

  1. Integrate Scrapy
  2. Integrate Puppeteer
  3. General Python spider
  4. General Node.js spider
© 2020 Crawlab, Made by Crawlab-Team all right reserved,powered by Gitbook该文件最后修改时间: 2020-10-05 06:00:14

results matching ""

    No results matching ""

    results matching ""

      No results matching ""