This benchmark suite demonstrates the MASSIVE memory and token savings achieved by using TOON (Token-Oriented Object Notation) compared to JSON for structured data.

NEW: TOON now supports direct conversion from Pydantic models with encode_pydantic() and decode_to_pydantic() functions!

Tested across 50 diverse, real-world datasets:

╔═══════════════════════════════════════════════════════════════╗
║                                                               ║
║                    ⚡ TOON DELIVERS ⚡                         ║
║                                                               ║
║     📉  63.9% SMALLER file sizes                              ║
║     📉  54.1% FEWER tokens for LLM APIs                       ║
║     💾  35.81KB total memory saved                            ║
║     🎯  10,735 total tokens saved                             ║
║                                                               ║
║                 💰 COST SAVINGS 💰                             ║
║     $2,147 per million API requests                           ║
║     $5,408 per billion tokens                                 ║
║                                                               ║
╚═══════════════════════════════════════════════════════════════╝

👉 SEE FULL RESULTS | 98% of datasets achieve 40%+ savings!

TOON achieves remarkable reductions compared to JSON:

For LLM API usage at typical GPT-4 pricing ($10/1M tokens):

Across 50 diverse, real-world datasets:

🔥 EXCELLENT (≥60% savings):  30 datasets (60%)
✅ GOOD (40-60% savings):     19 datasets (38%)
📊 MODERATE (<40% savings):    1 dataset  (2%)

98% of tested datasets achieve 40%+ savings!

View complete results for all 50 datasets →

TOON excels particularly with:

• Tabular data (e.g., database results, inventory): up to 73.4% reduction
• Uniform arrays (e.g., ML training data): up to 63.4% token savings
• Structured records (e.g., e-commerce products): 66.3% size reduction
• Analytics data (surveys, metrics): consistently 65-73% savings
• E-commerce (products, reviews): consistently 66-69% savings

Only 1 out of 50 datasets (2%) achieved <40% savings:

• Deeply nested objects with non-uniform structure (39% savings for Shipping Tracking)

Even in the worst case, TOON maintains readability while providing significant savings.

# Install the package with dependencies
pip install -e .

# tiktoken is required for token counting
pip install tiktoken

# Pydantic is optional but recommended for model validation
pip install pydantic

# Run the complete benchmark suite (tests all 50 datasets)
python benchmark/run_all.py

# Compare file sizes and token counts (all 50 datasets)
python benchmark/compare_formats.py

# Measure memory usage (subset of datasets)
python benchmark/memory_benchmark.py

The benchmark tests 50 diverse, real-world datasets including:

• E-commerce (products, orders, reviews, inventory)
• Databases (query results, employee records)
• APIs (responses, logs, requests)
• Analytics (metrics, surveys, A/B tests)
• IoT (sensor data, time series)
• Social media (posts, profiles, comments)
• Finance (transactions, stock data)
• And much more!

• compare_formats.py - Compares JSON vs TOON across 50 datasets
• memory_benchmark.py - Measures actual memory consumption
• sample_datasets.py - Collection of 50 realistic test datasets
• run_all.py - Executes all benchmarks and generates summary
• RESULTS.md - Complete detailed results for all 50 datasets

JSON (1,607 bytes, 552 tokens):

{
  "products": [
    {
      "id": 1001,
      "sku": "LAP-001",
      "name": "Gaming Laptop",
      "price": 1299.99,
      "stock": 45,
      "category": "Electronics"
    },
    ...
  ]
}

TOON (542 bytes, 231 tokens):

products[10]{id,sku,name,price,stock,category}:
  1001,LAP-001,Gaming Laptop,1299.99,45,Electronics
  1002,MOU-042,Wireless Mouse,29.99,234,Accessories
  ...

Savings: 66.3% size, 58.2% tokens

JSON (1,552 bytes, 481 tokens):

{
  "query": "SELECT * FROM employees WHERE department = 'Engineering'",
  "rows": [
    {
      "emp_id": 1001,
      "name": "Alice Johnson",
      "department": "Engineering",
      "salary": 95000,
      "start_date": "2020-03-15",
      "remote": true
    },
    ...
  ]
}

TOON (582 bytes, 209 tokens):

query: SELECT * FROM employees WHERE department = 'Engineering'
rows[8]{emp_id,name,department,salary,start_date,remote}:
  1001,Alice Johnson,Engineering,95000,2020-03-15,true
  1002,Bob Smith,Engineering,105000,2019-07-22,false
  ...

Savings: 62.5% size, 56.5% tokens

JSON repeats keys for every object:

[
  {"id": 1, "name": "A", "price": 10},
  {"id": 2, "name": "B", "price": 20}
]

TOON declares headers once:

[2]{id,name,price}:
  1,A,10
  2,B,20

JSON requires:

• Braces: { }
• Brackets: [ ]
• Quotes around all keys: "key"
• Quotes around string values: "value"
• Commas everywhere: ,

TOON uses:

• Indentation for structure (like YAML)
• Colons for key-value pairs: key: value
• Quotes only when necessary
• Headers for uniform arrays

TOON automatically:

• Detects when quotes aren't needed
• Uses compact array format for uniform data
• Preserves types (numbers, booleans, null)
• Maintains human readability

While TOON is slightly slower than native JSON (which is implemented in C), the difference is negligible for typical use cases:

• JSON encoding: ~0.005-0.06 ms per operation
• TOON encoding: ~0.03-0.57 ms per operation
• JSON decoding: ~0.004-0.05 ms per operation
• TOON decoding: ~0.04-0.62 ms per operation

Bottom line: For 99% of use cases, the performance difference is imperceptible, and the memory/token savings far outweigh the minimal overhead.

Understanding the tradeoff between cost savings and performance overhead is crucial for optimization decisions. Here's the Pareto frontier analysis:

The following table shows datasets on or near the Pareto frontier, offering optimal tradeoffs:

Efficiency Score = (Token Savings %) / (Encoding Overhead Factor)

Performance Overhead (encoding time relative to JSON)
    ↑
18x │                                      ◆ Deeply Nested
    │                                         Objects
16x │
    │
14x │                              ⭐ Tabular Data
    │                                 (Survey, ML)
12x │                          ⭐ E-commerce
    │                             Products
10x │                      ⭐ Analytics
    │                         Data
 8x │                  ⭐ Database
    │                     Results
 6x │              ⭐ API
    │                 Responses
 4x │          ◆ Simple
    │             Nested
 2x │
    │
 0x └────────────────────────────────────────────────────────→
    30%   40%   50%   60%   70%   80%   Cost Savings (token reduction %)

⭐ = Pareto-optimal (best tradeoff: high savings, acceptable overhead)
◆ = Sub-optimal (dominated: either low savings or excessive overhead)

Pareto Frontier ≈ Points where no other dataset offers both:
  • Higher cost savings AND lower performance overhead

Efficiency Ratio = Token Savings % / Performance Overhead Factor (higher is better)

Key Insight: Datasets on the Pareto frontier achieve 4.7-7.0% token savings per unit of performance overhead. Structured/tabular data consistently outperforms nested objects.

Use TOON when savings justify overhead (Pareto frontier):

• ✅ Tabular/structured data: 60-70% savings, 10-15x overhead → ROI: Excellent
• ✅ Repeated LLM calls: Even 1ms overhead saves $$$ on tokens over time
• ✅ Large payloads (>1KB): Serialization overhead amortized over size
• ✅ Bandwidth-constrained: Network transfer time >> encoding time

Reconsider TOON when (below Pareto frontier):

• ⚠️ Real-time systems: Microsecond latency requirements
• ⚠️ Single small payloads: <100 bytes where overhead dominates
• ⚠️ Highly irregular data: <40% savings with same overhead

When does TOON pay for itself?

For a typical e-commerce product catalog (1.6KB JSON → 542B TOON):

• Encoding overhead: +0.5ms
• Token savings: 321 tokens (58.2%)
• Cost savings per request: $0.00321 @ $10/1M tokens
• Break-even point: ~156 requests to recover 1 second of CPU time

Verdict: TOON is Pareto-optimal for almost all LLM API use cases, as token costs dominate CPU costs by 1000x.

1. LLM API Payloads - Reduce token costs by 50%+
2. Database Query Results - Compact tabular data representation
3. Analytics/Metrics Data - Efficient time-series and aggregate data
4. ML Training Data - Compress feature vectors and labels
5. E-commerce Catalogs - Product listings with uniform structure
6. Inventory Systems - Large collections of similar items
7. Log Aggregation - Structured log entries with common fields

1. Highly Irregular Data - Where no two objects share the same structure
2. Maximum Compatibility - When you need universal JSON tool support
3. Extreme Performance - When microseconds matter (though TOON is still fast)

Our benchmarks use:

• tiktoken for accurate GPT-4 token counting
• Real-world datasets representing common use cases
• Multiple iterations (1,000+) for performance measurements
• Actual memory profiling using sys.getsizeof

All benchmark code is open source and can be reviewed in this directory.

• TOON Format Specification
• Main README - Includes Pydantic integration documentation
• Pydantic Examples - Direct conversion from Pydantic models
• Python Package

Found a dataset where TOON could perform better? Want to add more benchmarks?

1. Add your dataset to sample_datasets.py
2. Run the benchmarks
3. Submit a PR with your findings!

┌────────────────────────────────────────────────────────────┐
│  JSON vs TOON - Average Savings (50 Datasets)             │
├────────────────────────────────────────────────────────────┤
│                                                            │
│  File Size:     ████████████████████████████████░░ 63.9%  │
│  Tokens:        ███████████████████████████░░░░░░░ 54.1%  │
│  API Costs:     ███████████████████████████░░░░░░░ 54.1%  │
│                                                            │
│  Best Case:     █████████████████████████████████░ 73.4%  │
│  (Survey Responses)                                        │
│                                                            │
│  98% of datasets achieve 40%+ savings                      │
│  60% of datasets achieve 60%+ savings                      │
│                                                            │
└────────────────────────────────────────────────────────────┘

💾 Remember: Every byte saved is a token saved, and every token saved is money saved when working with LLM APIs!

🎉 With 63.9% size reduction and 54.1% token reduction across 50 diverse datasets, TOON delivers massive, consistent savings for real-world structured data!