Two-Pac

Logo
GameTree
Menu
Game

Inspiration 💡

We love playing games, and one of our childhood memories was going to Arcades. As technology advances, our favourite pastimes have evolved alongside it. This inspired us to create Two-Pac, a retro-gamified photo album and a classic arcade game inspired by Pac-Man that takes its player on an interactive journey through their photo gallery and crafts narratives from the images of their favourite memories.

What it does 🎯

Our application gets 3 images from user's photo gallery, and evenly split them into 4 pieces (image shards) and place them into the Pac-Man maze. The goal of the player is to successfully collect the shards from the maze without being caught by the ghosts, allowing the player to relive the memories and experiences captured in the photo.
Upon collecting a shard (of which there are 4 in each level), a part of the narrative associated with the image the shard belongs to is revealed to the player. Once the player collects all the shards on the current level or is caught by a ghost, the player's victory/loss is noted and the player progresses to the next level. The progression of the player is dynamic and depends on whether they win or lose on different levels.
Upon completion of the levels, Two-Pac assembles the complete picture from the shards the player has collected through their victories and losses and reveals the formed narrative.

How we built it 🦾

We built Two-Pac using Python, PyGame, Vision Transformers, and the Cohere API.

Our project heavily relies on a base Pac-Man game, which we implemented in PyGame. To put a spin on classic Pac-Man and turn it into a gamified photo album, we had to introduce the concept of an "image shard" to the game, for which we added additional code and novel sprites.

A fundamental aspect of our project is its ability to automatically generate descriptions and narratives from the players images. First, Two-Pac uses SOTA Vision Transformers to automatically generate captions for the images and then, using the captions, utilizes Cohere's Generation API to craft a dynamic and immersive narrative that links the memories of the images together.

Since the player's progression through the game impacts the narrative they face, we utilize the Game Tree data structure to capture the various possible game states. The nodes of the Game Tree represent states, while the edges represent the player winning/losing a level. Using the Cohere Generation API, we populate the different paths of the Game Tree, ensuring consistency between generated narratives by conditioning on the generated narrative of images in the ancestor states in the data structure.

Upon initialization of the game, the player's images are decomposed into 4 sub-images and stored as image shards. Upon completion of the game, Two-Pac uses stored information to recreate the (in?)complete image and narrative using the shards the player collects.

Challenges we ran into 🚦

There were a few challenges that were particularly difficult.

One of our biggest challenges was with trying to incorporate the Cohere API with our Game Tree generation. We had a lot of difficulty trying to ensure that the LLM's output was consistent with the desired format. None of us had any prior experience with Prompt Engineering so we had to read guides and talk to the Cohere mentors for help. Ultimately, a Reddit post resolved our issues (apparently, if you want the LLM output to be parse-able you just need to mention it in the the input).

Another big challenge was with linking different parts of the program together. Different team members worked on different parts in isolation, and trying to make everyone's code adaptable with everyone else's was surprisingly difficult.

Accomplishments that we're proud of 🌟

We did not know each other before the hackathon, and we take pride in the fact that we were able to get together, make a plan, and stick to it while supporting each other. Additionally, we find it really cool that we bridged the disconnect between classical AI and modern AI by using Game Trees, LLMs, and Vision Transformers to bring new life to a classic game.

What we learned 🔍

Cindy: During the process of making Two-Pac, I learned to make use of all the available resources. And I also realize that being a good programmer is more than just coding, there are still a lot ahead of me for me to learn, such as operating systems.

Duc: I learned the skill of combining individual features together to form a bigger product. As a result, I also learned the importance of writing clean + concise code since ~~trash~~ poorly written code causes a lot of issues when debugging.

Rohan: I learned to take in a lot of new code and find ways to add on new features. This means that writing clean, decoupled code is really important.

Rudraksh: This was my first time having to work with Prompt Engineering. I learned that its advantageous to forgo conversational instructions and give concise and specific details about the format of the input and desired output.

What's next for Two-Pac ❔

There are many features that we want to implement in the future. One feature we wanted to include but didn't have the time to implement was "song of the year" - using AI, we sought to estimate the year the photo was taken to play some music from that year when one of that image's shards was collected.