File System

Crawlab uses SeaweedFS as its distributed file system.

Below is how it works with files synchronization between the master node and worker nodes.