AniFriend

AniFriend is an AI-powered dating simulator that combines Live2D animated characters, emotional intelligence, and voice synthesis for romantic interactions.

Comment 1

About the Project: AniFriend: Valentine's Day Dating Simulator

Inspiration

Valentine's Day can be a lonely time for many people. Whether you're single, separated from loved ones, or simply want to experience something unique, we wanted to create an interactive experience that brings warmth and connection to everyone. The inspiration came from the Japanese visual novel and dating simulator genre, combined with modern AI technology to create truly dynamic, emotionally intelligent conversations.

We wanted to prove that AI companions could provide meaningful interactions while maintaining appropriate boundaries and healthy relationship dynamics. The goal wasn't just to create a chatbot, but to craft an immersive experience complete with:

  • Live2D animated characters that react with genuine expressions
  • Voice synthesis that brings text to life
  • Progressive relationship systems that reward meaningful interaction
  • Interactive cafe dates that simulate real romantic experiences

What We Learned

Technical Skills

  1. Live2D Integration with Web Technologies

    • Integrating Cubism SDK with PIXI.js for real-time 2D animation
    • Managing model expressions and animations dynamically
    • Handling multiple character models with different rigs and expressions

  2. AI Prompt Engineering

    • Crafting system prompts that maintain consistent character personalities
    • Implementing dynamic affection systems that influence AI behavior
    • Balancing creativity with appropriate content boundaries

  3. Multi-Modal AI Integration

    • Combining text generation (Google Gemini)
    • Text-to-speech synthesis (ElevenLabs)
    • Speech-to-text transcription for voice input
    • Synchronizing all these systems seamlessly

  4. State Management in React

    • Managing complex state with affection levels, milestones, and chat history
    • Implementing real-time emotion analysis and expression updates
    • Handling async operations across multiple API services

Soft Skills

  • Emotional Design: Understanding how UI/UX affects user emotional engagement
  • Narrative Design: Creating compelling character backstories and interaction patterns
  • Ethical AI: Establishing boundaries for AI interactions to promote healthy behaviors

How We Built It

Architecture Overview

The project is built with a modern web stack:

Next.js 14 (App Router) + TypeScript
    ↓
┌─────────────────────────────────────────┐
│  Frontend Layer                         │
│  - React Components                     │
│  - PIXI.js + Live2D Display             │
│  - State Management (useState/useEffect)│
└─────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────┐
│  API Layer (Next.js Route Handlers)     │
│  - /api/chat    → Gemini AI + TTS       │
│  - /api/stt     → Speech-to-Text        │
│  - /api/tts     → Text-to-Speech        │
└─────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────┐
│  External Services                      │
│  - Google Gemini API (Text Generation)  │
│  - ElevenLabs API (Voice Synthesis)     │
└─────────────────────────────────────────┘

Key Components

  1. ModelCanvas Component

    • Loads and renders Live2D models using pixi-live2d-display
    • Dynamically switches between characters (Arisa/Chitose)
    • Maps emotion strings to model expressions
    • Handles scaling and positioning for responsive display

  2. Chat System

    • Maintains conversation history for context-aware responses
    • Implements affection calculation: \(A_{new} = A_{current} + (B \times M_e)\)
      • Where \(A = affection \ level \)
      • \(B= base \ change (5 \ points) \)
      • \(M_e = emotion \ multiplier (-3 \ to \ +3)\)
    • Progressive milestone system at thresholds: \({25, 50, 75, 100}\)

  3. Emotion Analysis System

    • Uses regex pattern matching to detect emotional indicators in AI responses
    • Different emotion mappings for each character:
      • Arisa: Smile, Angry, Sad, Surprised, Normal
      • Chitose: Smile, Angry, Sad, Surprised, Normal, Blushing
    • Real-time expression updates synchronized with text

  4. Cafe Date Feature

    • Unlocks at affection level ≥ 50
    • Virtual currency system for ordering
    • Menu categories with item costs
    • Receipt generation with timestamps

Development Process

graph LR
    A[Character Design] --> B[Live2D Integration]
    B --> C[AI Personality Crafting]
    C --> D[Voice Synthesis]
    D --> E[Affection System]
    E --> F[Cafe Date Feature]
    F --> G[Polish & Testing]

Challenges We Faced

1. Live2D SDK Compatibility Issues

Problem: The Live2D Cubism SDK requires specific loading order and has conflicts with Next.js SSR.

Solution:

  • Disabled SSR for the ModelCanvas component using dynamic import with ssr: false
  • Manually loaded the Cubism Core script before the display library
  • Exposed PIXI globally before Live2D initialization
// This order is critical!
await loadScript("/live2d.min.js");
await loadScript("https://cubism.live2d.com/.../live2dcubismcore.min.js");
const { Live2DModel } = await import('pixi-live2d-display');

2. Expression Mapping Complexity

Problem: Different Live2D models have different expression file names and structures.

Solution: Created character-specific expression mappings:

const chitoseExpressions = {
  'Angry': 'Angry',
  'Sad': 'Sad',
  'Smile': 'Smile',
  // ... maps emotion concepts to actual .exp3.json files
};

3. Emotion Detection Accuracy

Problem: Simple keyword matching produced inconsistent results.

Solution: Implemented priority-based regex patterns with context awareness:

  • Check for specific emotion indicators in order of uniqueness
  • Use word boundaries (\b) to avoid false positives
  • Consider punctuation patterns (e.g., [!?]{2,} for surprise)

4. Audio Streaming Performance

Problem: ElevenLabs TTS returns streaming data that needed buffering.

Solution: Implemented chunk-based buffering and Base64 encoding:

const chunks: Uint8Array[] = [];
const reader = audio.getReader();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  chunks.push(value);
}
const audioBuffer = Buffer.concat(chunks);

5. Affection System Balance

Problem: Initial affection changes were too rapid, making progression feel unrealistic.

Solution:

  • Tuned multipliers based on extensive testing
  • Made affection changes proportional to emotional response intensity
  • Implemented character-specific multipliers (Chitose has gentler progression)

6. State Synchronization

Problem: Expression changes, audio playback, and text updates needed perfect timing.

Solution: Used React's useEffect dependencies carefully and implemented callback-based sequencing for critical updates.

About the Side Project: Button Tag

Track: Useless Hacks Button Tag is an interactive React web application built with TypeScript and Parcel that features a playful cat-and-mouse game where users attempt to click an elusive button that moves around the screen to avoid the cursor. The app includes two game modes (escape and chase), an inventory system for placing decorative "stick" shapes on the canvas, and defensive abilities like shrinking and becoming translucent that activate when the button is cornered. Built with a minimalist setup using Parcel as the build tool, the project demonstrates creative UI interactions and game mechanics in a browser-based environment.

Built With

Share this project:

Updates