An intelligent document analysis tool that uses Large Language Models to generate comprehensive, hierarchical mindmaps from any text document.

Interactive Mermaid Diagram	Markdown Outline
Interactive HTML visualization with expandable nodes	Hierarchical Markdown output for easy reference

The Mindmap Generator is a sophisticated document analysis system that extracts the core concepts, relationships, and details from text documents and organizes them into intuitive, hierarchical mindmaps. Unlike simple text summarization, this tool:

• Intelligently adapts to different document types (legal, technical, scientific, narrative, etc.)
• Creates multi-level hierarchical representations (topics, subtopics, details)
• Ensures factual accuracy by verifying against the source document
• Eliminates redundant or overlapping concepts
• Generates outputs in multiple formats (Mermaid syntax, HTML, Markdown)
• See an example Mermaid Diagram in the live editor here

The system is built to work with a variety of LLM providers (OpenAI, Anthropic/Claude, DeepSeek, Google Gemini) and optimizes for both cost-efficiency and output quality.

You can read a detailed blog post about the making of this project and the various challenges and design considerations that went into the design and engineering of it here.

• Document Type Detection: Automatically adapts extraction strategies based on document type
• Hierarchical Content Extraction: Builds three-level hierarchies (topics → subtopics → details)
• Reality Checking: Verifies generated content against the source document to prevent confabulation
• Duplicate Detection: Uses both fuzzy matching and semantic similarity to avoid redundancy
• Multi-format Output:
  • Mermaid mindmap syntax
  • Interactive HTML with Mermaid rendering
  • Markdown outline
• Cost Optimization: Designed to work efficiently with value-priced LLMs
• Rich Logging: Detailed, color-coded progress tracking (see an example here)

1. Install Pyenv and Python 3.12 (if needed):

# Install Pyenv and python 3.12 if needed and then use it to create venv:
if ! command -v pyenv &> /dev/null; then
    sudo apt-get update
    sudo apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev \
    libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
    xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git

    git clone https://github.com/pyenv/pyenv.git ~/.pyenv
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
    echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
    echo 'eval "$(pyenv init --path)"' >> ~/.zshrc
    source ~/.zshrc
fi
cd ~/.pyenv && git pull && cd -
pyenv install 3.12

2. Set up the project:

# Use pyenv to create virtual environment:
git clone https://github.com/Dicklesworthstone/mindmap-generator    
cd mindmap-generator          
pyenv local 3.12
python -m venv venv
source venv/bin/activate
python -m pip install --upgrade pip
python -m pip install wheel
python -m pip install --upgrade setuptools wheel
pip install -r requirements.txt

3. Set up your environment:

Create a .env file with your API keys:

OPENAI_API_KEY="your-key"
ANTHROPIC_API_KEY="your-key" 
DEEPSEEK_API_KEY="your-key"
GEMINI_API_KEY="your-key"
API_PROVIDER="OPENAI"  # Options: "OPENAI", "CLAUDE", "DEEPSEEK", or "GEMINI"

1. Edit the mindmap_generator.py file to specify your input document:

input_file = "sample_input_document_as_markdown__durnovo_memo.md"  # <-- Change

2. Run the generator:

python mindmap_generator.py

3. Find your generated outputs in the mindmap_outputs directory:
   • {filename}_mindmap__{provider}.txt - Mermaid syntax
   • {filename}_mindmap__{provider}.html - Interactive HTML visualization
   • {filename}_mindmap_outline__{provider}.md - Markdown outline

This repository includes a fascinating historical document as a test case - the famous Durnovo memo from 1914, which remarkably predicted World War I and the Russian Revolution. For more about this incredible document, see my article about it here.

The Durnovo Memorandum was written by Pyotr Durnovo, a Russian statesman, to Tsar Nicholas II in February 1914 - months before the outbreak of World War I. With astonishing prescience, Durnovo warned about:

• The inevitability of war between Germany and Russia if European tensions continued
• How such a war would lead to social revolution in Russia
• The collapse of monarchies across Europe
• The specific dangers Russia faced in a prolonged European conflict

The memo has been hailed as one of the most accurate political predictions in modern history, making it an excellent test document for our mindmap generator.

We've processed this document using all four supported LLM providers to demonstrate how each handles the complex historical content. The results showcase each provider's strengths and unique approaches to concept extraction and organization.

OpenAI's model produced a concise, well-structured mindmap with clear hierarchical organization:

• Mermaid Syntax (2.8 KB)
• Interactive HTML (5.7 KB)
• Markdown Outline (2.5 KB)

GPT-4o-mini excels at producing compact, efficient mindmaps that capture essential concepts without redundancy. Its output is characterized by clear categorization and precise language.

Claude produced a more detailed mindmap with richer contextual information:

• Mermaid Syntax (4.1 KB)
• Interactive HTML (7.3 KB)
• Markdown Outline (3.8 KB)

Claude's approach tends to include more nuanced historical context and captures subtle relationships between concepts. Its output is particularly strong in preserving the memo's analytical reasoning.

DeepSeek generated the most comprehensive mindmap with extensive subtopics and details:

• Mermaid Syntax (9.0 KB)
• Interactive HTML (15 KB)
• Markdown Outline (8.4 KB)

DeepSeek's output is notable for its thoroughness and depth of analysis. It extracts more subtleties from the document but occasionally at the cost of some redundancy.

Gemini created a balanced mindmap with strong thematic organization:

• Mermaid Syntax (5.5 KB)
• Interactive HTML (9.6 KB)
• Markdown Outline (5.0 KB)

Gemini's approach focuses on thematic coherence, with particularly strong extraction of geopolitical concepts and causal relationships between events.

This multi-provider approach reveals interesting patterns:

1. Content Organization Differences: Each provider structures the document's concepts differently, revealing their unique approaches to conceptual organization
2. Detail Granularity Variance: The level of detail varies significantly, with DeepSeek providing the most comprehensive extraction and OpenAI the most concise
3. Emoji Selection Patterns: Each model has distinct tendencies in selecting representative emojis for concepts
4. Historical Context Sensitivity: Models differ in how they handle historical context, with Claude showing particular strength in preserving historical nuance
5. Structured Knowledge Representation: The differences highlight various approaches to knowledge organization from the different AI research teams

The sample outputs serve as both demonstration of the tool's capabilities and an interesting comparative study of how different LLM providers approach the same complex historical document.

Unlike traditional LLM applications that operate linearly, the Mindmap Generator employs a sophisticated, non-linear architecture that resembles an undirected graph of exploration. Here's an in-depth look at how the system works:

Traditional LLM applications typically follow a simple pattern:

Input → LLM Prompt → Output

Or perhaps a pipeline:

Input → LLM Prompt 1 → Output 1 → LLM Prompt 2 → Output 2 → Final Result

The Mindmap Generator, however, operates as a multi-dimensional exploration system, where:

1. Multiple parallel processes explore different aspects of the document simultaneously
2. Feedback loops evaluate the quality and uniqueness of extracted information
3. Heuristic-guided decisions determine when to explore deeper or stop exploration
4. Verification mechanisms ensure factual accuracy throughout

This approach allows the system to efficiently navigate the vast conceptual space of the document while maintaining coherence and accuracy.

The system begins by analyzing a sample of the document to determine its fundamental type, which guides subsequent extraction strategies:

• Technical documents focus on system components, interfaces, and implementations
• Scientific documents emphasize research methodologies, results, and theoretical frameworks
• Narrative documents highlight plot elements, character development, and thematic elements
• Business documents extract strategic initiatives, market opportunities, and implementation plans
• Legal documents identify legal principles, rights, obligations, and procedural requirements
• Academic documents focus on theoretical frameworks, scholarly arguments, and evidence
• Instructional documents extract learning objectives, skill development, and assessment methods

Each document type has specialized prompt templates optimized for extracting the most relevant information. Rather than a simple classification, the system uses specific detection heuristics that identify key indicators of document structure and purpose.

A key innovation is how the system handles large documents:

1. Overlapping Chunk Creation: Documents are divided into manageable chunks with deliberate overlap to preserve context at boundaries
2. Boundary Optimization: Chunk boundaries are adjusted to coincide with natural breaks (e.g., end of sentences) rather than arbitrary character counts
3. Context Preservation: The overlap between chunks ensures that concepts that span chunk boundaries aren't fragmented
4. Progressive Exploration: Chunks are processed in a way that builds cumulative understanding of the document

This approach solves the fundamental limitation of LLM context windows while ensuring no important information is lost at chunk boundaries.

The topic extraction process employs a sophisticated multi-stage approach:

1. Parallel Initial Extraction: Multiple chunks are analyzed simultaneously to identify potential topics
2. Frequency Analysis: Topics that appear across multiple chunks receive higher significance
3. Consolidation Phase: Similar topics are merged into cohesive, distinct concepts
4. Semantic Deduplication: Multiple similarity detection methods (including LLM-based semantic analysis) ensure topics are genuinely distinct
5. Importance Ranking: Topics are weighted based on document coverage, frequency, and semantic significance

This multi-phase approach ensures that the extracted topics provide balanced coverage of the document's content while avoiding redundancy or over-fragmentation.

The system employs an adaptive strategy that optimizes resource usage:

1. Priority-Based Processing: More important topics receive deeper exploration
2. Diminishing Returns Detection: The system recognizes when additional processing yields minimal new insights
3. Breadth-Depth Balancing: The exploration automatically adjusts between breadth (covering more topics) and depth (exploring topics in greater detail) based on document complexity
4. Completion Thresholds: Sophisticated heuristics determine when sufficient information has been extracted

This adaptive approach ensures that the system allocates computational resources efficiently, focusing effort where it will provide the most value.

One of the most challenging aspects of mindmap generation is eliminating redundancy while preserving unique information. The system employs a multi-layered approach:

1. Textual Similarity: Basic string matching identifies obvious duplicates
2. Fuzzy Matching: Fuzzy string algorithms detect near-duplicate content with variations
3. Token-Based Analysis: Comparing token patterns identifies structural similarities
4. LLM-Based Semantic Analysis: For conceptually similar but textually different content, the system uses the LLM itself to evaluate semantic similarity
5. Hierarchical Redundancy Checking: Redundancy is checked within levels (e.g., between topics) and across levels (e.g., between a topic and a subtopic)

This comprehensive approach prevents the mindmap from containing repetitive information while ensuring nothing important is lost.

To prevent confabulation (the generation of factually incorrect information), the system implements a sophisticated verification mechanism:

1. Content Verification: Each generated node is compared against the source document to ensure it's either explicitly stated or logically derivable
2. Confidence Scoring: Verification results include confidence metrics that influence node inclusion
3. Structural Preservation: The system balances factual accuracy with maintaining a coherent mindmap structure
4. Verification Statistics: Detailed metrics track verification success rates across different node types

This reality check ensures that the mindmap remains a faithful representation of the source document, even when dealing with complex or abstract content.

The system enriches the visual representation of the mindmap through intelligent emoji selection:

1. Context-Aware Selection: Emojis are chosen based on the semantic content of each node
2. Hierarchical Differentiation: Different node types (topics, subtopics, details) use visually distinct emoji styles
3. Importance Indicators: Special markers indicate the importance level of details
4. Persistent Caching: Emoji selections are cached to ensure consistency across generations
5. Fallback Hierarchy: If optimal emoji selection fails, the system follows a thoughtful fallback strategy

This visual enhancement makes the mindmap more engaging and easier to navigate, with visual cues that communicate additional meaning.

A significant challenge was making the system work effectively with more affordable LLM models:

• Prompt Optimization: Each prompt is carefully crafted to be concise yet comprehensive
• Context Limitation: The system deliberately limits context to prevent cognitive overload
• Task Isolation: Complex tasks are broken down into simpler, focused sub-tasks
• Progressive Refinement: Results are incrementally improved rather than attempting perfect outputs in one step
• Error Recovery: The system detects and handles cases where LLM outputs are inconsistent or low-quality

These strategies allow the system to leverage less expensive models while maintaining high-quality outputs.

The system employs a sophisticated asynchronous architecture:

1. Task Orchestration: Complex dependency graphs manage the flow of tasks
2. Semaphore-Based Rate Limiting: Prevents overwhelming API rate limits
3. Exponential Backoff with Jitter: Intelligent retry logic for handling failures
4. Cooperative Task Scheduling: Efficient resource utilization across concurrent operations
5. Dynamic Priority Adjustment: More important tasks receive processing priority

This asynchronous design dramatically improves throughput while maintaining control over execution flow.

The system employs sophisticated heuristics to ensure balanced content extraction:

1. Minimum Coverage Requirements: Ensures sufficient breadth across the document
2. Distribution Balancing: Prevents over-representation of specific sections
3. Hierarchical Proportion Control: Maintains appropriate ratios between topics, subtopics, and details
4. Importance-Weighted Selection: More significant content receives greater representation
5. Content Type Diversity: Ensures a mix of conceptual, factual, and supporting information

These heuristics ensure that the final mindmap provides a balanced representation of the document's content.

The system incorporates multiple layers of error handling:

1. Graceful Degradation: The system continues operating effectively even when some components fail
2. Result Validation: All LLM outputs are validated before being incorporated
3. Fallback Strategies: Alternative approaches are employed when primary methods fail
4. State Preservation: Intermediate results are cached to prevent lost work
5. Comprehensive Logging: Detailed error information facilitates debugging and improvement

This resilience ensures reliable operation even with unreliable LLM responses or API limitations.

The system implements detailed token usage tracking:

1. Category-Based Tracking: Usage is broken down by functional categories
2. Cost Calculation: Token counts are converted to cost estimates based on provider pricing
3. Comparative Analysis: Usage patterns are analyzed to identify optimization opportunities
4. Trend Monitoring: Usage patterns over time help identify shifts in performance

This tracking provides transparency and supports ongoing optimization efforts. You can see an example of what it looks like here.

Several strategies minimize costs while maintaining output quality:

1. Early Stopping: Processing halts when sufficient quality is achieved
2. Tiered Processing: Less expensive models handle simpler tasks
3. Caching: Frequently used results are cached to prevent redundant API calls
4. Content Batching: Multiple items are processed in single API calls where possible
5. Similarity Pre-filtering: Cheaper computational methods filter candidates before expensive LLM-based comparisons

These strategies significantly reduce the total cost of generating comprehensive mindmaps.

The system provides rich performance visualization:

1. Color-Coded Progress: Visual indicators show processing status at a glance
2. Hierarchical Metrics: Performance is tracked at multiple levels of granularity
3. Completion Ratios: Progress toward completion is continuously updated
4. Cost Projections: Running cost estimates provide financial transparency
5. Quality Indicators: Verification rates and confidence scores indicate output reliability

These visual tools make it easy to monitor long-running processes and understand system behavior.

The system can iteratively improve mindmaps through targeted refinement:

1. Quality Assessment: Existing mindmaps are evaluated for balance and coverage
2. Targeted Enhancement: Specific areas are identified for improvement
3. Differential Processing: Only areas requiring enhancement are reprocessed
4. Consolidation: New insights are integrated with existing content
5. Before/After Comparison: Changes are tracked to evaluate improvement

This approach allows efficient enhancement of existing mindmaps without complete regeneration.

The system is designed to work with multiple LLM providers:

1. Provider-Specific Optimization: Prompt templates are tailored to each provider's strengths
2. Unified Interface: A consistent interface abstracts provider differences
3. Dynamic Selection: The optimal provider can be chosen based on task requirements
4. Cost Balancing: Tasks are allocated to minimize overall cost across providers
5. Fallback Chains: If one provider fails, the system can automatically retry with alternatives

This flexibility ensures the system remains viable as the LLM landscape evolves.

Different document types receive specialized processing:

1. Technical Documents: Function diagrams and dependency mappings
2. Scientific Documents: Methodology flowcharts and result visualizations
3. Narrative Documents: Character relationship maps and plot progression
4. Business Documents: Strategic frameworks and implementation timelines
5. Legal Documents: Requirement hierarchies and procedural workflows

These specialized enhancements maximize the value of the generated mindmaps for different document types.

mindmap
    ((📄))
        ((🏛️ Constitutional Framework))
            (📜 Historical Context)
                [🔸 The memo begins by examining the historical context of constitutional interpretation]
                [🔹 References to the Federalist Papers and early American political thought]
                [🔸 Discussion of how the Constitution was designed to balance power]
            (⚖️ Separation of Powers)
                [♦️ Detailed analysis of the three branches of government and their distinct roles]
                [🔸 Examination of checks and balances between branches]
                [🔹 Historical examples of power struggles between branches]

# Constitutional Framework

## Historical Context
The memo begins by examining the historical context of constitutional interpretation
References to the Federalist Papers and early American political thought
Discussion of how the Constitution was designed to balance power

## Separation of Powers
Detailed analysis of the three branches of government and their distinct roles
Examination of checks and balances between branches
Historical examples of power struggles between branches

The Mindmap Generator excels in various scenarios:

• Literature Review: Quickly understand the key concepts and relationships in academic papers
• Thesis Organization: Structure complex research findings into coherent frameworks
• Concept Mapping: Visualize theoretical relationships across multiple sources

• Strategic Document Analysis: Extract actionable insights from lengthy business reports
• Competitive Research: Organize information about market competitors
• Policy Implementation: Break down complex policies into implementable components

• Case Brief Creation: Distill lengthy legal opinions into structured hierarchies
• Regulatory Compliance: Map complex regulatory requirements
• Contract Review: Identify key obligations and provisions in legal documents

• Curriculum Development: Organize educational materials into logical learning paths
• Study Guide Creation: Generate comprehensive study guides from textbooks
• Knowledge Mapping: Create visual representations of subject matter domains

• Architecture Understanding: Map complex technical systems
• API Documentation: Organize endpoint functionality into logical groupings
• System Requirements: Structure complex requirement documents

MIT License

If you find this project useful, you might also be interested in my other open-source projects:

• LLM Aided OCR
• Your Source to Prompt
• Swiss Army Llama
• Fast Vector Similarity
• PPP Loan Fraud Analysis