aNode

The best solution to challenge 3. Decentralised, highly scalable, and highly reliable cloud infrastructure.

Comment 1

About aNode

What it does

  1. Real-time data sharing. The data will be shared throughout the cluster in near real-time after one aNode received some data from a publisher, and push it to the subscribers via WebSocket connections.
  2. Decentralized data storage. With the help of the Apache Cassandra, a decentralized database, we've implemented decentralized data storage with every node holds the same replica of the data. Besides, decentralization is the reason why we don't use MySQL, Kafka, Etcd or PostgreSQL -- they will break our truly decentralized P2P architecture.
  3. Highly scalable. It's easy to scale up by just starting new docker containers when you need a stronger performance.

How we built it

Decentralized P2P Architecture

We designed a decentralized architecture based on the Gossip protocol, which is a robust and scalable way to manage information dissemination in a decentralized system, with reliable and efficient communication among multiple aNode instances.

The diagram of this architecture is as follows:

Decentralized Real-time Data Transfer (Performance)

Once a node receives new data, it stores it to Cassandra and, after a certain period, gossips it to other nodes. This process continues, ensuring rapid and robust dissemination of data. Eventually, the data reaches all or most of the nodes in the network, leading to a consistent state among them. It can also handle high throughput, especially in scenarios where updates are frequent and need to be transferred quickly. Besides, by only sending information to a few nodes at a time, our system can be more bandwidth-efficient compared to broadcasting to all nodes.

A new data is able to received by all nodes in a time complexity of $O(logN)$ (N is the number of nodes in the cluster).

Resilience

The Gossip protocol is inherently fault-tolerant. If some nodes fail or are unreachable, the information still propagates through other paths in the network. Moreover, this approach scales well with the number of nodes, as each node only communicates with a few others at any given time.

Unlike centralized systems, there is no single point of control or failure in our decentralized architecture. This network is also unstructured, which means the system is high availability and participation tolerance (comply with AP in the CAP theorem).

Scalability

The architecture baed on the Gossip protocol is inherently scalable due to the nature of decentralization. As the number of nodes increases, the system naturally adapts because each node only communicates with a few others at any given time. This means that the overall cluster does not become significantly more congested as it grows. Besides, the load of disseminating data is distributed among all nodes, preventing any single node from becoming a bottleneck.

Challenges we ran into

  1. Design a decentralized infrastructure. It's really tricky for me to design a decentralized architecture at first because I used to focused on centralized Web2 distribute applications (like microservices) in the past. This is a tough challenge for me to learn how to build a "true" infrastructure (i.e., a distributed system with the least dependencies as possible).
  2. Duplicate messages. A node may receive the same message more than once due to the inner working for the Gossip protocol, which is hard to detect and breaks the integrity of the data. We added a timestamp property to the data, which indicates the fetch time of this data. While updating the local state of a node, for the same data source, we only do the update when the timestamp of the data is later than the current one.
  3. Implementing the Gossip protocol. It's really tricky to implementing such a complex distributed protocol like the Gossip protocol. After studying some infrastructures that also use the Gossip protocol, I found that we can use some libraries (like memberlist) to implement the Gossip protocol and reducing development costs.

Accomplishments that we're proud of

  1. Completely meets the requirement of challenge 3. Challenge 3 mainly requires real-time data sharing, easy to use, high scalability, cost-effective, and no setup fees. Decentralized is not a requirement, but is a plus. aNode completely meets those requirements under a decentralized architecture. The following is a break-down that shows how we done this.
  2. Truly decentralized P2P architecture. Architecture is one of the hardest things to change in software development. While other teams are still designing centralized, Web2 architecture, we have completed a completely decentralized, P2P Web3 architecture. With the help of the Gossip protocol, it's naturally highly scalable and highly fault tolerant, which complies to AP in the CAP theorem, and is still able to provide all the functionalities with only one node.
  3. Containerized and easy to use. The whole back-end is able to start by a single docker-compose up -d command, and it's also quite simple to scale up by simply adding more services in the docker-compose.yml. That means you can set up a large aNode cluster in minutes, not hours or days, and it's easy to transfer to larger container management systems like Kubernetes and Istio.
  4. No setup fees. The container of aNode just contains the instance itself, and can be deployed anywhere there is a Docker installed. If it's deployed on public clouds, user will only pay for the computing (caused by aNode itself) and storage (caused by Apache Cassandra) with no additional costs.

What we learned

  1. How to build a decentralized architecture. I had no idea when I first heard about decentralized architecture. However, after 10 days of searching and designing, I designed a completely decentralized, highly scalable and P2P architecture that comply with AP in the CAP theorem, and learned a lot about software design and decentralized architecture.
  2. How to implement the Gossip protocol. Our decentralized architecture is based on the Gossip protocol, which is tricky for us. However, with the help of the memberlist library provided by HashiCorp, which is based on the Gossip protocol, we completed the cluster management and real-time data sharing successfully.
  3. How to execute efficient team management. Although we are not perfect at team management, we still learned a lot from this hackathon. I now have a deeper understanding of project management and team management, and can lead a small team to complete a project on my own.

What's next for aNode

  1. Continue developing. aNode is more than just a solution to this hackathon -- it's an open-source software. With the passion to cloud infrastructures and open-source, we will continue developing and maintaining it.
  2. Use decentralized storage (like Tableland or IPFS). We are now using Apache Cassandra, which is decentralized but not really suitable for Web3 developments. We will transfer to decentralized storages in the future.
  3. More front-end pages. We're focused on the completely decentralized architecture while hacking, and somehow ignored the front-end pages. Our front-end pages are just some kind of demo, and is not fully completed. We will build a user-focus and powerful front-end in the future.
  4. Pull (via RESTful APIs) and plugins support. For data fetching, we are now only support push (via WebSocket API /publish). However, pull is another important method that have no side effects to the data source. Besides, we can make such data fetching methods plugins, and let the user choose how to collect data.
  5. GraphQL support. GraphQL is a powerful method for querying, and it's easy to support via some libraries.
  6. Middlewares support. It's really common that user may want to do some modification to their data after the data was fetched. We will implement this via middlewares, and give users a chance to develop and deploy their own middlewares.

Team Acknowledgement

In the successful completion of our hackathon project, the distinct contributions of Kiko (Qingru Liu), Qui Dam, and Hailey (Huirong Ma) have been instrumental. I (Ada (Jingrong Mu, [email protected])) couldn't complete this project without the help of them.

Kiko ([email protected]), as the project manager, ensured seamless communication with sponsors and led the team with agility, especially critical during periods when I was away from Sydney.

Qui Dam ([email protected])'s proficiency as a full-stack developer solidified the technical backbone of our project, addressing complex challenges with ease.

Hailey ([email protected]), our dedicated front-end developer, worked remotely from China, showcasing remarkable adaptability and skill in bringing our project's interface to life.

Their collective efforts not only enhanced our project's quality but also embodied the spirit of collaboration and innovation.

Built With

Share this project:

Updates