Valy | Devpost

Valy
Example
Architecture

Inspiration

As a Machine Learning Engineer, my daily work involves solving complex problems with data and AI. I have a passion for Generative AI and have been playing VALORANT since its Beta release. This hackathon presented the perfect opportunity to combine my professional skills with something I love. It allowed me to challenge myself, push the boundaries of what I know, and immerse myself in a problem space that I’m genuinely passionate about.

I highly recommend reading the documentation from the repository, which has more detail and information regarding the application.

What it does

Valy is an AI-powered assistant that helps VALORANT esports managers with tasks such as scouting top players, analyzing player roles, and constructing the best possible team compositions. Valy can:

Identify the best players for specific roles like Duelist or Controller.
Analyze historical data to suggest optimal team compositions.
Retrieve detailed stats on individual players, including performance metrics like kills per round (KPR), assists per round (APR), and more.
Provide insights on the top players using specific agents.

By simplifying complex data, Valy offers managers actionable insights to make smarter, data-driven decisions for team-building and strategy planning.

How I built it

I built Valy using a serverless architecture to ensure scalability and cost-efficiency. Here’s an overview of the tech stack:

Amazon S3 stores the raw and processed esports data.
AWS Glue processes the JSON data into Parquet format for efficient querying.
Amazon Athena powers the retrieval of dynamic player stats and team compositions.
The core intelligence is provided by Amazon Bedrock's Haiku 3 LLM, enhanced with Cohere embeddings for semantic search stored in Pinecone.
AWS Lambda functions dynamically process and retrieve player data, while FastAPI provides a chat interface for real-time interactions.
The system is deployed using AppRunner for fast and scalable access, making sure users can interact with Valy efficiently.

Methodology

Bedrock-Powered Digital Assistant

Valy's assistant is powered by Amazon Bedrock, using Claude Haiku 3 as the core LLM agent. The agent was configured to perform specific tasks through an action group that integrates with a Lambda function. This function contains five primary actions:

get_top_players_by_role: Retrieves the top players for a specific role (e.g., Duelist, Controller).
get_best_combination_of_roles: Analyzes past game data to suggest the best role composition for building a successful team based on wins.
get_historical_player_stats: Fetches historical performance statistics for a particular player, including data across all agents.
get_top_players_by_agent: Identifies the top players using a specific agent.
get_meta_composition_by_map: Retrieves the meta composition for a specific map.

Knowledge Base and Embeddings

Valy leverages a Bedrock Knowledge Base that is connected to Pinecone for storing vector embeddings, which were generated using Bedrock's Cohere (English v3) embeddings. This knowledge base contains:

Detailed information about VALORANT agents and their abilities, sourced from the Valorant Fandom Wiki.
Biography of professional players, sourced from Liquipedia Valorant.
Impact Metric Description, created by me.

These embeddings allow Valy to perform semantic searches, retrieving the most relevant and accurate information about players, agents, and strategies in response to user queries.

Data Processing and Analysis

The system processes match data through several stages:

Data Preparation:
- Ingests JSON format match data from VCT tournaments
- Processes game events including:
  - damageEvent
  - gameDecided
  - playerDied
  - configuration
- Converts data to Parquet format using AWS Glue
- Creates optimized tables and views in Athena
Player Impact Metrics: Custom impact formulas are used for each role (Duelist, Controller, Initiator, Sentinel). These metrics measure player impact based on key performance indicators, factoring in the number of rounds played.

Formulas for Player Impact by Role:

Duelist:

Impact = (0.35 * KPR + 0.25 * FKDR + 0.15 * APR + 0.15 * KDR + 0.10 * FKPR) * (1 - e^(-Rounds Played / Avg Rounds Played))

Controller:

Impact = (0.20 * KPR + 0.15 * FKDR + 0.30 * APR + 0.20 * KDR + 0.15 * FKPR) * (1 - e^(-Rounds Played / Avg Rounds Played))

Initiator:

Impact = (0.25 * KPR + 0.20 * FKDR + 0.30 * APR + 0.10 * KDR + 0.15 * FKPR) * (1 - e^(-Rounds Played / Avg Rounds Played))

Sentinel:

Impact = (0.20 * KPR + 0.15 * FKDR + 0.25 * APR + 0.25 * KDR + 0.15 * FKPR) * (1 - e^(-Rounds Played / Avg Rounds Played))

Explanation of Metrics:

KPR: Kills per round
FKDR: First kill/death ratio
APR: Assists per round
KDR: Kill/death ratio
FKPR: First kills per round
Rounds Played: The total number of rounds played by the player
Avg Rounds Played: The average number of rounds played by all players in the dataset

How the Formula Works:

Each formula assigns different weights to the relevant performance metrics for each role. The sum of the weighted metrics provides the player's performance score for the respective role. This score is then adjusted by the number of rounds played to normalize the score over time and reduce noise from players with fewer rounds played.

The exponential factor (1 - e^(Rounds Played / Avg Rounds Played) ensures that players with more rounds played are weighted higher, giving a more reliable representation of their performance. This prevents players with only a few outstanding rounds from being ranked higher than players with consistently good performance over a larger number of games.

Challenges I ran into

Several challenges arose during the development of Valy, including:

Inconsistent historical data: The structure of esports event data has evolved over time, making it difficult to align older datasets with newer ones. This required extensive data normalization.
Player identity management: Players often use different accounts across tournaments or regions, making it hard to track consistent player stats.
Limited ability impact data: For non-duelist roles, the lack of detailed ability effectiveness metrics made it challenging to assess certain players’ performance fully.
Lack of centralized esports documentation: Esports-specific guidelines and rules are scattered, which limited the AI’s ability to provide rule-based team compositions.

Round 2: What would I do if I didn't have throttling issues

In Round 2, new prompts were introduced, and based on previous tests in Round 1, most of them should be addressed effectively. Here are my hypothetical suggestions for further improvements.

What recent performances or statistics justify the inclusion of this player in the team?

R: In Round 1, Valy has access to each player's historical performance, which can be filtered by tournament and year, allowing him to retrieve recent performance data. To enhance this, a new table (player_scores) with finer detail was added to display each player's game-by-game scores in chronological order.
If this player were unavailable, who would be a suitable replacement and why?

R: From Round 1, Valy can retrieve a list of players by role, so if a primary player (e.g., TOP (k)) is unavailable, he can identify the next in line (TOP (k+1)) along with their performance history. No additional changes are required.
Which maps would this team composition excel in? Why?

R: This question couldn’t be directly addressed in Round 1 due to limited data. To solve it, we introduced a structured data regarding current meta compositions per map. Combined with historical performance data for each player, this information helps identify which meta composition the team can leverage to excel on specific maps.
Which player would take a leadership (IGL) role in this team? Why?

R: From Round 1, Valy has access to each player's profile, which includes career details that help identify leadership potential. This allows him to recommend a suitable In-Game Leader (IGL) based on experience or relevant attributes.
Role-specific metrics and justification (Duelists, Sentinels, Controllers, Initiators). Ex. How effective is this player in initiating fights and securing entry kills?

R: In Round 1, Impact Metrics by Role were developed to rank players by weighting their stats according to the impact within their specific role. However, the descriptions of these metrics were not initially provided to Valy, so they have since been added as unstructured data to ensure clarity in interpretation.

Considerations

All structured data added will be accessible via Lambda, while unstructured data will be stored in the knowledge base.
Minimal adjustments to agent prompts may also be needed to ensure they know how to effectively use this new information.

Accomplishments that I'm proud of

This is my first AWS project, and I'm proud to have successfully navigated the complexities of AWS services.
I was able to manage and process complex and large esports data to generate meaningful insights for team management.
I implemented everything I know about AI, LLMs, and agents to create an AI assistant that understands and responds to esports-specific queries.
I developed custom impact metrics for different player roles, providing a nuanced evaluation of player performance.
The entire system is built with scalability in mind, allowing for continuous improvements through CI/CD pipelines and Terraform, making it easy to add new data sources and expand functionalities.

What I learned

Throughout this project, I learned:

The importance of data consistency and quality when working with esports data, and how inconsistencies can affect AI models.
How to optimize AWS services like Glue, Lambda, and Athena to handle dynamic and large-scale data processing.
The complexities of tracking player identity across tournaments due to changing accounts and aliases.
The significance of developing role-specific metrics to fairly evaluate players and account for the number of rounds played, which ensures balanced comparisons.

What's next for Valy

In the future, I plan to further enhance Valy by:

Real-time data processing: Integrating live event data to allow for up-to-the-minute analysis and insights during ongoing matches.
Expanded player profiles: Incorporating additional player data like nationality, age, and team history, along with more comprehensive statistics on ability effectiveness.
Improved agent intelligence: Upgrading Valy's core AI model to incorporate multi-stage prompting and decision trees for more complex queries, potentially using even more powerful models like GPT-4.
Temporal weighting for performance data: Applying a weighting system that prioritizes recent performances more heavily, ensuring that Valy’s recommendations reflect current form over older data.
Adding more unstructured information: With Terraform and CI/CD pipelines, I can easily continue to add new types of data and features, such as expanded player profiles, deeper stats analysis, or even new query capabilities to retrieve more specific statistics.
Better player identity management: Implementing a more robust system for tracking players across different accounts and regions, making the system more reliable for scouting.