SeaweedFS

SeaweedFS is an open-source distributed file system which is good at hosting small files. As Crawlab is a distributed crawler management platform, SeaweedFS serves as the medium for synchronizing files such as scripts and program files between nodes and instances.

For more details, please refer to the wiki on GitHub.