If you want to see your spider results on the Crawlab interface, you need to integrate your spider with Crawlab.
The following are prerequisites for spider integration:
- You need to set result set ;
- You need to write the data in a database with Crawlab, for example, 'crawlab_test';
- You need to write the result back to the specified dataset in the spider ('CRAWLAB_COLLECTION'), and assign a value in the 'task_id'('CRAWLAB_TASK_ID') field.
Here are some examples of spider integration: