StreamerPedia

Logo
StreamerPedia Extension running on Twitch
StreamerPedia Interface
Multiple Chats with Separate Memory

Inspiration

Twitch has millions of streamers and every one of them has built a world around their content, inside jokes, community members, tournament results, and strategies that took years to develop. But none of it is searchable. It lives in chat logs that vanish in 2 weeks, VODs nobody watches twice, and Twitter threads buried under years of posts. This is very overwhelming for small creators and for new viewers.

We felt this most in niche gaming communities. If you watch a Minecraft speedrunner, there are entire categories of strategies and route names that any longtime viewer knows by heart, but a new viewer is completely lost. If you ask ChatGPT, it either hallucinates or draws a blank. The knowledge exists, it is just not where AI can find it. We wanted to fix that and in turn empower new viewers to join established communities without feeling like outsiders, while giving creators a way to preserve their history automatically, turning fleeting stream moments into a lasting, searchable knowledge base.

What it does

StreamerPedia is the DeepWiki for Twitch streamers. You install the Chrome extension, open any stream, and ask anything about the streamer, their community, their strategies, or their history. You get accurate, streaming answers grounded in that streamer's ACTUAL content.

The system learns from many sources of streamer knowledge: YouTube transcripts, Twitter, Twitch VODs and clips, and live stream audio transcribed locally on your device using Whisper. The more people use it on a stream, the smarter it gets, because every viewer contributes to the same shared knowledge base.

How we built it

We built a full-stack pipeline across three layers. The Chrome extension captures live chat over Twitch IRC and transcribes stream audio locally in a Web Worker using Whisper, so audio never leaves your machine. A signal detector scores sliding chat windows for significance and sends bundles to our FastAPI backend, where an LLM extracts structured knowledge facts, filters by confidence, and stores them as 768-dimensional Gemini embeddings in a Supabase pgvector database.

At query time, the API runs a pgvector similarity search over the streamer's knowledge base, pulls live stream context from the extension, and gives the LLM a Tavily web search tool it can call if stored knowledge is not enough. Answers stream back token by token through a React side panel that persists across browser sessions, with multiple chat tabs each scoped to a different streamer.

Challenges we ran into

1. Extracting live audio within Chrome's security model Getting access to a tab's raw audio stream for local processing is difficult due to browser sandboxing. We had to implement an offscreen document pattern to capture the audio stream via chrome.tabCapture, and then pipe it into a dedicated Web Worker running a quantized version of Whisper. Orchestrating the message passing between the content script, the background service worker, and the offscreen processor without causing memory leaks or UI lag was our biggest technical hurdle.

2. Figuring out what is important in real time We couldn't just process every second of the stream as running an LLM on 6 hours of gameplay would be very expensive and fill the database with junk. We had to build a client-side Signal Detector that monitors chat velocity, emote spikes, and question repetition to decide when a moment is worth capturing. Tuning these heuristics to capture "lore moments" without triggering on every random "LOL" spam required a lot of trial and error with adaptive sliding windows.

3. Structuring unstructured streamer knowledge We quickly learned that simply dumping raw chat logs into a vector database leads to poor RAG performance. Streamer lore is specific: facts, events, inside jokes, and metrics all look different. We had to design a flexible schema for our database and implement a confidence scoring system (filtering for >0.80 confidence). This ensures we only store high-quality facts rather than random chat noise, preventing the AI from hallucinating based on a single spam message.

4. Managing multi-streamer context and state Building a specific Side Panel UI that supports multiple chat tabs, where each tab is locked to a different streamer's knowledge base, required complex state management. We had to decouple the "active recording" stream (what you are watching) from the "active chat" session (who you are asking about). This allows a user to watch xQc while asking questions in a separate tab about Ludwig, with the backend correctly routing the context and RAG queries to the right knowledge partition.

Accomplishments that we're proud of

We are really proud that the whole thing actually works end to end. Audio transcription runs fully locally, knowledge gets extracted and stored in real time as you watch, and questions come back with genuinely accurate, stream-specific answers that a generic AI model cannot generate. The live context pipeline in particular, where what the streamer is saying right now feeds directly into your answer, feels like something new.

We are also proud of how cleanly the architecture held up across all the moving parts. The extension, the API, the vector database, and the scraper all work together without friction, and the product feels polished enough that you could hand it to anyone and they would know immediately what to do with it.

What we learned

We learned a lot about the limits of Chrome extension APIs and how to work within them creatively. The offscreen document pattern for audio capture, the Web Worker for Whisper, the service worker as an orchestrator that never touches audio directly, none of that was obvious upfront and figuring it out taught us a lot about how browsers actually manage memory and execution contexts.

On the AI side, we learned that RAG quality depends almost entirely on how well you structure what goes into the database. The typed knowledge schema, the confidence filtering, the per-type prompt templates for injection, all of that made a significant difference in answer quality compared to just dumping raw text into a vector store.

What's next for StreamerPedia

The most exciting next step is making the knowledge base collaborative and public. Right now every streamer gets their own private knowledge base built by whoever happens to be watching. We want to make that database open, so the community can verify, correct, and contribute to it directly, turning StreamerPedia into something truly community-owned.

We also want to expand beyond Twitch. The same pipeline works for any live content platform, and the niche community problem we are solving is not unique to gaming. Sports, music, politics, education, anywhere people gather around a creator and build shared knowledge that mainstream AI has never seen, StreamerPedia can help.

Tech Stack

Languages

TypeScript (Chrome Extension): The entire extension frontend, background workers, and offscreen audio processors are written in strict TypeScript.
Python 3.12 (Backend API): The backend logic, scraping orchestration, and RAG pipeline are built in Python.

Frameworks and Libraries

Frontend/Extension: React, Vite, vite-plugin-web-extension, @xenova/transformers (local Whisper), react-markdown.
Backend: FastAPI (async web framework), uvicorn, pydantic (data validation), asyncio.
AI & Data: openai (SDK for LLM calls), google-generativeai (for Gemini embeddings), supabase (Postgres client), pgvector (vector similarity search).
Scraping: youtube-transcript-api, twitch-downloader-cli, httpx (async HTTP client).

Platforms & Cloud Services

Supabase: Our primary database provider. We rely heavily on its PostgreSQL offering for relational data and pgvector extension for storing and querying 768-dimensional embeddings.
Vercel: Hosts our public-facing FastAPI backend (serverless deployment).
OpenAI Compatible Endpoint: We use an OpenAI-compatible endpoint to serve gpt-oss-120b for high-quality knowledge extraction and question answering.
Google Gemini: Used for generating text embeddings (text-embedding-001).
Tavily: An AI-optimized search engine API used for real-time web retrieval during RAG.