AIm Assist

For more in-depth information about the technical process and the decision-making behind it, please check out the .pdf that we attached! You can also access it here. Our code repository can be found here. Please request for access to the repository by sending a message to [email protected]

Inspiration

We are three friends who have known each other since high school over a decade ago. Having watched competitive League of Legends - and, subsequently, Valorant, for over a decade, competitive Esports has always been a place that allowed us to maintain our connection. None of us had played tactical shooters (or even shooters at all!!) prior to Valorant, but thanks to the goodwill that Riot built up, we decided to give Valorant a try and got invested in the VCT. While some of us do not have as much free time to partake in gaming as frequently as when we were younger, we all take the competitive leagues of our favourite games very seriously - we theorize about the tactical aspects, read too much into tweets to fantasize about potential rosters. We scream in elation (or cry in agony) based on the performance of the teams with our favourite players. We were fortunate enough to participate in the Global Esports Ranking Hackathon - and managed to utilize all the hours we devoted to following the competitive league to come up with a unique solution. While we weren't able to win the top prize, we gained valuable experience using the AWS suite of services, and our main feedback to Devpost, Amazon, and Riot was that we wanted to do another one. Luckily, we were able to find out about this event due to the social connections we made from the previous event and we are hoping we can come up with an exciting and unique solution!

What it does

AIm Assist is the one-stop shop for all your competitive Valorant team-building needs. It generates effective Valorant rosters based on your questions and utilizes data that our team painstakingly munged. It employs a Retrieval-Augmented Generation(RAG) to feed our rich document data to our AI Agents, which synthesize those documents with our high-quality data. The creation of novel composite statistics, AIm Score and IGL Score, reduced the computational load of our AI system, allowing it to process user queries and provide excellent outputs in a timely manner. The app also has other additional features, such as an intuitive chat interface, document uploads, chat history, and a data viewer to analyze the statistics of AIm Assist's proposed roster.

How we built it

AIm Assist uses LangChain and LangGraph to generate a graph-based framework that directs how a user query will interact with different AI elements and utilizes Streamlit to provide an interactive interface. The system uses a FAISS vector store to efficiently retrieve relevant documents for interpreting player statistics and roster formation strategies. An agent is integrated to query our database for specific data to supplement our document data with objective statistics. Amazon Bedrock serves as the backbone through the use of its foundational model as well as the ability to scale our product to a more reliable and function-rich service. The project is hosted on an AWS EC2 instance, with AWS S3 handling data storage for scalability and reliability.

Challenges we ran into

Data Challenges
One major challenge occurred when we were processing the data, and a big part of it was trying to integrate Riot's game data with the community-sourced data. Since Riot game data had numerical identifiers, their "player handle" data were allowed to be inconsistent, often using a myriad of different variations of player's usernames (e.g. FNS, FiNESSE). There are also scenarios where many players had the same player handle but were from other regions, so they had to be differentiated in the game data (e.g. mel1, meL, Mel). Community-sourced data don't have a numerical identifier, so we had to use a player handle to match and merge the data. This led to a lot of missing data and errors. Luckily, there was enough context in the game data to resolve many of the mistakes and figure out specifically which player was which, but it did require a lot of manual input and tinkering.

AWS Bedrock Challenges
We originally tried to create our AI solution utilizing only Amazon Bedrock, but we ran into access issues and errors when we created policies to activate our knowledge base. We spent so long debugging the administrative issues that we didn't feel confident that we could show off our analytical and creative solutions due to spending too much time on tooling. Even the elements we managed to get working were just costing us money (such as AOSS) without us actually utilizing it in our solution. We decided we would create a minimum viable product using langchain and once we got a working prototype we would then port our solution to Amazon Bedrock. We really wanted to emphasize our creativity and passion for competitive Valorant, and we felt that our inexperience with AI tools was hampering our ability to demonstrate that. Funnily enough, this approach allowed us to create and finish a working prototype a few days before the end of the hackathon. Around that time our team, along with many others, were running into errors with throttling in the foundational models and because our POC was originally decoupled from Bedrock, we still had a stable way to test our models.

Throttling Errors
Another challenge was that as we approached the deadline for the project, there were issues with throttling errors. This error prevented us from being able to test and make improvements to our prompts using Amazon Bedrock. To circumvent these errors, we decided to use the Anthropic API in order to continue testing and improving the answers to our prompts. As such, for our Chatbot, there is an option that allows users to Use Anthropic Model. This would enable judges and users to check out the creative and in-depth answers AIm Assists provides to the prompts for the competition. It should be noted here that there is an issue with the number of tokens we have available for the Anthropic Model.

Accomplishments that we're proud of

We are proud to employ mixed-methods research and a rigorous statistical framework for a subject matter of interest with a dearth of novel independent research. We surveyed Esports professionals and thematically analyzed Valorant Content (e.g., podcasts) for our qualitative analysis. For the quantitative approach, we engineered our own novel statistics, and we found that they tracked well against highly regarded data sources (VLR Rating)

In the context of how we started this project and what inspired it, we are very impressed by the AIm Score and the IGL score that we meticulously created based on the game data. This metric allowed us to objectively and systematically rank players for what previously were hypotheticals in our discussions. It also allowed us to have more meaningful conversations about player performance and roster creations (and, of course, who is more correct based on actual data to resolve friendly disputes). We had a lot of fun discussing the game, many important players, and how managers may look at team rosters based on the results of the project we created for the Hackathon.

We were also extremely proud of the technical implementation of our AI chatbot. We went through long discussions to determine the most efficient system to use and the trade-offs between which steps to take to refine the LLM system and which LLM system we ultimately chose to use. This information can be found in the attached PDF file (AIm_Assist_write_up.pdf).

What we learned

While our AI system works well and delivers impressive outputs, the true magic behind its performance isn't the complexity of its architecture. Instead, the real driver of performance lies in the data feeding into it. Getting relevant responses from the model is like a careful balancing act on a tightrope - just feeding large amounts of data and hoping that the model returns something insightful can lead to hallucinations or dead ends in responses. Instead, we shifted our approach by utilizing a streamlined data strategy. We used document data to build a small but potent corpus that summarizes how to assess a player. To complement this, we also processed the data and built statistical features that reflect what professional analysts use to evaluate players and teams (two main ones being AIm Score and IGL Score). These composite metrics serve as a guide for the AI, significantly simplifying their task.

Without this data strategy, we suspect the model and the agents would carry much heavier computational loads, leading to higher hallucination risks or misfiring with incomplete contexts.

What's next for AIm Assist

We are delighted with how well the product turned out after a month of development! While the app is functional, we had a lot of ideas that we abandoned due to time constraints, but we may implement them in the following months! Here are a few that we were thinking about:

Better LLMs selection - currently, we are utilizing Claude Sonnet 3.5 for stability, but there are some model invocations in our system that could be served equally well without the advanced reasoning (and the computational overhead that comes with it).
Improved UI - we scraped together a usable UI in a few days for ease of use. We want to utilize UI UX best practices to improve our front end solution.
Expanded Features - we want to utilize AI to create data visualizations so that we can better utilize the blank spaces currently in our app.
Video Search - would take our player roster and search up their gameplay or tournament performance - similar to what a service from Perplexity.AI provides. This is an excellent way to visualize how a player plays beyond just the numbers.
Currency-based Roster Creation - can be used to develop more realistic rosters. For instance, it can be unrealistic to just get the best players on a roster due to salary costs. Being able to assign currency points to players can lead to more balanced teams, and could lead to other interesting Apps like Fantasy VCT.