FileDAG

FileDAG is a decentralized storage solution based on IPFS. It will provide high-quality and efficient IPFS Pin Service and customized network service based on customer requirements.

Comment

Inspiration

An NFT project team wanted to store their data on the IPFS website, however, they met a few technical problems and asked our team for help.

What it does

FileDAG is a decentralized storage solution based on IPFS. It will provide high-quality and efficient IPFS Pin Service and customized network service based on customer requirements.

Go-Datastore-Cluster is one of the most significant parts of FileDAG, which aims to gather FileDAG’s distributed key-value datastores into a cluster, to increase the function of data split, and improve the management of data in key-value datastores. It’s the storage layer of FileDAG.

In certain scenarios, the user wants to store a large dataset on IPFS and backup a few copies on Filecoin. At this point, Go-Graphsplit, as a component of FileDAG, could be the right tool used for splitting large datasets into graph slices fit for making deals on the Filecoin network, which will streamline data retrieval. We will keep improving it to match FileDAG’s functions better.

How we built it

FileDAG:

  • IPFS for data sharing
  • Gin for API framework
  • PostgreSQL for service data storage
  • GRPC for internal service communication

Datastore-Cluster:

  • Hash Slots for data sharding
  • Libp2p for data and information sharing
  • CRDT for the consensus model

Graphsplit:

  • IPLD for build or split file directory tree

Challenges we ran into

Reconstructed the technical architecture of key-value datastores of IPFS cluster into Go-Datastore-Cluster.

Accomplishments that we're proud of

An easy-to-use front-end UI and a more developer-friendly API service will minimize the technical thresholds and make IPFS usable. With the FileDAG Website, the user can upload files (under 10 GiB) or IPFS CID to pin the data they need. For developers who want to use FileDAG’s API to pin data, they just need to create an API key, and then use API to enjoy our pin service. All files can be managed in the File Management Center with more actions, such as unpinning, editing metadata or pin policy, and deleting data from the user’s account.

What we learned

IPFS cluster, Libp2p protocol, and how to match FileDAG with Web3.Storage.

What's next for FileDAG

The SDK of FileDAG will be available shortly soon.

Roadmap of Go-Datastore-Cluster:

  • data sharding and hash slots maintaining
  • communication module based on libp2p between data nodes
  • consensus module build-up
  • hash slots re-allocate and re-balance strategy
  • data migration after hash slots re-allocate or re-balance to support dynamically adding or removing nodes
  • authentication and data management
Share this project:

Updates