Inspiration
When using Chat GPT, I frequently question what it still remembers about my current conversation. To combat this, I needlessly paste older content to make sure it is included in the context window. I wanted a tool to help me identify what is missing from my context window. Our goal is to provide a window into the black box of Chat GPT.
What it does
Token Bin is a Chrome extension that tracks the number of tokens used in a conversation with ChatGPT, comparing this to the backend model's token limit. This allows users to identify missing context within the model's inherent context window. By highlighting and reintegrating crucial omitted details in subsequent requests, Token Bin ensures that ChatGPT maintains a comprehensive understanding of the conversation. It also provides valuable metrics like token consumption, model information, and latency data, enabling a more nuanced analysis of the interaction.
How we built it
We built Token Bin by integrating a Python server with JavaScript service workers. The extension parses the DOM after each user request, securely transmitting the conversation history for token analysis. Our JavaScript service workers render the 'missing context' in the extension’s copy bin. Additionally, we intercept POST requests from the GPT server to the Chrome source module, allowing us to assess crucial token information and analyze latency. Our stack is a blend of JavaScript for frontend dynamics and Python Flask for backend stability.
Challenges we ran into
One challenge we ran into was intercepting the network requests that Chat GPT sends. We sifted through all of the requests to find valuable information to give the user. We also had difficulty knowing where to split the context. Since it is not immediately available how many tokens are in a given chunk of text, we needed to perform a binary search to find where to split the text.
Accomplishments that we're proud of
Since this our first hackathon for all of us, we are very proud to have made a functional tool that we would use! We are also proud of the seamless way to reintegrate context back into the conversation.
What we learned
Our journey with Token Bin has been incredibly enlightening. We delved deep into the intricacies of OpenAI’s management of GPT's context windows, transforming a common user issue into an opportunity for innovation and learning. This project not only solved a practical problem but also expanded our understanding of how advanced language models like GPT function.
What's next for Token Bin
The future of Token Bin is exciting. We are embarking on a phase focused on extensive data collection to refine our understanding of 'missing context.' With a current dataset that's relatively small, our priority is to expand it, allowing for more precise tuning of our algorithms. We're also exploring ways to get GPT to summarize the missing context, moving beyond plain text reintegrations. This will be a game-changer for developers and power users who rely on GPT in their workflows and face challenges related to context management.
Log in or sign up for Devpost to join the conversation.