Spider integration

If you want to see your spider results on the Crawlab interface, you need to integrate your spider with Crawlab.

The following are prerequisites for spider integration:

• You need to set result set ;
• You need to write the data in a database with Crawlab, for example, 'crawlab_test';
• You need to write the result back to the specified dataset in the spider ('CRAWLAB_COLLECTION'), and assign a value in the 'task_id'('CRAWLAB_TASK_ID') field.

Here are some examples of spider integration:

1. Integrate Scrapy
2. Integrate Puppeteer
3. General Python spider
4. General Node.js spider