DDiscord

Inspiration

Today I run a large discord bot If I wanted to deploy through a CI pipeline with the login rate limits and restart times to users I could only push code 10 times a day. We wanted to fix that, along with increasing reliability.

What it does

Our system uses a Kubernetes cluster to connect shards to discord, basically many small clients that each process a portion of the events. Those events then get ingested into Solace which handles distributing them to workers. Workers can be upgraded without discord even noticing. Workers are assigned the task of keeping the state of everything happening in the chat client up to date in Redis, allowing any of the scripts written to perform actions for the user fast and reliable access to up to date state.

How we built it

First we made a mono-repo and helper libraries, such as a wrapper around tho solace api. Then we wrote event ingest and event processing code to ingest the events into solace and process them into Redis Then we deployed to the google cloud, using Kubernetes to allow us to easily run multiple instances of the services in parallel.

Challenges we ran into

JS client library for solace provided a challenge in implementation within our requirements Debugging in a distributed cloud based environment provides a unique set of challenges and we ended up spending more time than we would have like building new containers and testing them.

Accomplishments that we're proud of

We got everything to run in the google cloud on Kubernetes or using managed services, not having to manage a single server or other than once for debugging type a bash command. The automatic shard management that assigns shards which servers to take care of is also neat.

What we learned

We learned a ton about distributed event systems and google cloud. It's hard to convey the amount of knowledge gained but basically we are now both confident using google cloud's Kubernetes / MemoryStore / Cloud Build / Registry. We also learned a ton about solace as an event distribution / messaging system in specific. How to use it, how to debug distributed event systems in a time efficient manner and more!

What's next for DDiscord

Next up is finding a way we can afford to keep running it. The google cloud is not cheap. Probably spinning up our own Kubernetes cluster with something like Rancher and some old server hardware and using something like circleCI / gitlab for our builds.

Built With

• ci
• docker
• docker-registry
• eris
• kubernetes
• node.js
• raincache
• redis
• snowtransfer
• solace