Skip to content

zacrh/xai

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Grok Context Notes

An alternative to Community Notes using Grok AI with synthetic jury consensus. The system identifies viral/inflammatory posts on X (Twitter), generates fact-checking notes using Grok, and evaluates them through a simulated jury of 356 real Community Notes raters to achieve cross-partisan consensus.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         X API Stream                            │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  PIPELINE (pipeline/)                                           │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │
│  │ ingest_posts│─▶│filter_posts │─▶│ check_grok  │              │
│  │ (stream)    │  │ (viral)     │  │ (classify)  │              │
│  └─────────────┘  └─────────────┘  └─────────────┘              │
│         │               │                │                      │
│         ▼               ▼                ▼                      │
│    posts.jsonl   controversial.jsonl  inflammatory.jsonl        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  JURY (jury/)                                                   │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐  │
│  │ Generate Notes  │─▶│ Jury Voting     │─▶│ Consensus Score │  │
│  │ (diverse)       │  │ (356 raters)    │  │ (threshold 0.6) │  │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  DISPLAY                                                        │
│  ┌────────────────────────┐    ┌───────────────────────────┐    │
│  │ frontend/              │    │ extension/                │    │
│  │ (X.com simulation)     │    │ (inject into live X.com)  │    │
│  └────────────────────────┘    └───────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Components

Pipeline (pipeline/)

Python scripts for ingesting and filtering X posts:

Script Description
ingest_posts.py Streams posts from X API filtered stream, saves to data/posts.jsonl
filter_posts.py Detects viral engagement spikes using statistical analysis (>1σ), outputs data/controversial.jsonl
check_grok.py Classifies controversial posts as inflammatory using Grok AI, outputs data/inflammatory.jsonl
load_db.py Loads posts to Supabase database
load_notes.py Loads jury evaluation results to Supabase database

Jury (jury/)

FastAPI backend for generating and evaluating fact-checking notes:

  • Note Generation: Creates 2 diverse candidate notes per tweet using Grok with web search
  • Jury Sampling: Stratified sampling from 356 real Community Notes raters (left/center/right)
  • Voting: Each juror votes on all candidates using in-context learning with their historical voting patterns
  • Consensus: Selects the note with highest agreement (threshold: 0.6), regenerates if needed
jury/
├── src/
│   ├── api/           # FastAPI endpoints
│   ├── generation/    # Note generation with Grok
│   ├── evaluation/    # Jury voting & consensus
│   └── stats/         # Metrics (brand safety, tribal rift)
└── data/              # Jury pool parquet (356 raters)

Frontend (frontend/)

React web application simulating X.com's interface with Grok context notes displayed:

  • 3-column layout (sidebar, feed, analytics widgets)
  • Displays posts with "Grok Added Context" notes
  • Analytics dashboard showing brand safety, tribal rift, and system health metrics

Extension (extension/)

Browser extension that injects Grok context notes into live X.com tabs:

  • Detects tweets as they enter viewport
  • Fetches notes from API and injects styled containers
  • Adds "Inflammatory" feed tab to X.com
  • Works on x.com and twitter.com

Data Generation (data_gen/)

Jupyter notebook for experiments with Community Notes data analysis.

Setup

Pipeline

cd pipeline
pip install xdk python-dotenv requests
python ingest_posts.py  # Start streaming

Jury

cd jury
uv sync
uv run uvicorn src.api.main:app --reload

Frontend

cd frontend
npm install
npm run dev

Extension

  1. Open Chrome → chrome://extensions
  2. Enable Developer Mode
  3. Click "Load unpacked" → select extension/ directory

Environment Variables

Create a .env file in the project root:

# X API
XAPI_BEARER_TOKEN=your_x_api_bearer_token

# Grok/xAI
XAI_API_KEY=your_xai_api_key

# Supabase
DB_URL=your_supabase_url
DB_KEY=your_supabase_key

Data Flow

  1. Ingest: Stream X posts matching filter rules → posts.jsonl
  2. Filter: Detect viral spikes (>1σ engagement) → controversial.jsonl
  3. Classify: Grok classifies inflammatory content → inflammatory.jsonl
  4. Generate: Jury system creates diverse fact-checking notes
  5. Vote: 356-rater jury evaluates notes for cross-partisan consensus
  6. Display: Frontend/extension shows accepted notes to users

API Endpoints

Jury API

  • POST /api/v1/evaluate-tweet - Full pipeline: generate notes → jury voting → consensus
  • GET /api/v1/health - Health check with jury pool status

Frontend API (Vercel)

  • GET /api/posts/feed - Paginated feed of posts with notes
  • GET /api/posts/[id]/notes - Get notes for a specific tweet

About

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors