Market Bubble

Witness the market bubbling in front of you. With air and water!

Comment

Inspiration

The infinity theme for this year's Durhack. Space -> bubbles. Artistic souls require artistic data representation. The market is such a fleeting thing - representing its changes with bubbles beautiful in all senses of the word. We were going for a unique, scalable, and reliable solution for data pipeline and visualisation.

What it does

We have developed a program that retrieves a large stream of Binance data from Finhub and uses Kafka (learned from the Marshall Wace workshop) to efficiently process that high-volume data stream and implement scalability. After analysing the transactions data stream, the program represents the frequency of a chosen transaction type using real-life bubbles. Bubbles!

How we built it

Many trials, many errors :)

Used Arduino ESP32 Wrover to control a stepper motor fixed around a water column of own design. We send bubbles up the water column in a frequency proportional to the volume of a chosen transaction (currency 1 to currency 2) type per 10 seconds.

Used Apache Kafka for efficient stream processing of such high volumes of data. Kafka also allowed for the scalability of our finance data stream, which is one of the big goals we were aiming for.

Challenges we ran into

Previous designs

Previous designs involved using a shutter door in between the water chamber (at the top of the column) and the air chamber (at the bottom of the column); and designing a one-way valve (because we surprisingly could not buy any at B&Q) to be placed at the bottom of the water-column only to be opened by the motor from underneath the design to let the air in when bubbles are to be generated. All of these were not working in the way intended. 12 hours into the project we landed onto the final design approach of squeezing the water column with a string (not a lanyard ;) ) pulled by a motor. 14 hours in - we got the first generated bubble. Faith had to be kept alive for many hours before we got this creation to work. But we did believe.

Kafka

Kafka turned out to be complex. Many aspects we learned about how it does work and even more about how to get it to work for us.

Engineering

We found out that building your ideas into the physical realm does not guarantee perfect execution in the very many first attempts. Eventually, we did get Market Bubble to work, which demonstrates a huge improvement in our Engineering Skills.

Accomplishments that we're proud of

  • Physics works!!
  • Bubbles are bubbling, trade rates are incoming.
  • Even got Kafka to work.

What we learned

  • water is a source of danger when it is next to exposed electronics;
  • how to use Kafka;
  • teamwork is a blessing;
  • Arduino has interesting logic for dealing with lights.

What's next for Market Bubble

  • waterproofing the whole design (case solution)
  • expanding to the lava lamp market

Built With

Share this project:

Updates