MongoDB

MongoDB is one of the most used NoSQL database which is schemaless and allows for rapid development because developers don't have to worry about database schema migration when data fields are changed.

The biggest reason of using MongoDB in Crawlab is its flexibility. As Crawlab is under fast development, most of the time the models and data fields need to be updated. Therefore, MongoDB is the best solution.

Crawled results data are also stored in MongoDB in the Community Edition. The database of results storage is the same as the operational data. If you would like to store results data in other types of databases (e.g. MySQL, PostgreSQL, ElasticSearch, etc.), or other databases or instances of MongoDB, you can check out the Professional Edition, which has more powerful features for production use.