This repository contains the implementation and experimental results for a multimodal subtitle translation system, focusing on Subtitle Recognition and Text-to-Speech (TTS).

• Model 1 (Baseline): Qwen/Qwen2-VL-2B-Instruct
• Model 2 (Enhanced): Qwen/Qwen3-VL-4B-Instruct
  • Note: Qwen3-VL-4B was run in a dedicated Python 3.11 environment.
• Model 3 (Alternative): OpenGVLab/InternVL2-4B
• Model 4 (Traditional): EasyOCR
• Method: Zero-shot Video-Language Understanding / OCR.
• Input: Video frames extracted at 1 frame/second.
• Output: Chinese subtitle text.

• Model 1 (Baseline): GPT-SoVITS v3
  • Method: Zero-shot Voice Cloning (Reference-based Synthesis).
• Model 2 (Comparison): F5-TTS
  • Method: Flow Matching (Zero-shot).
• Model 3 (Fallback): EdgeTTS
  • Method: API-based text-to-speech.
• Input: Source Chinese audio (for timbre reference) + Target Japanese text.
• Output: Japanese speech audio.

• Model 1: Qwen2.5-3B-Instruct (LoRA Fine-tuning)
  • Rank: 16, Alpha: 32, Quantization: 4-bit (NF4).
• Model 2: NLLB-200
  • Method: Machine Translation Baseline.
• Model 3: Qwen2.5-7B-Instruct
  • Method: Text-only LLM Zero-shot.

We evaluated the Optical Character Recognition (OCR) performance using several VLM and traditional baselines.

To address potential "missing subtitles" at low frame rates (1fps), we conducted a comparison on 11.mp4:

• 1 fps: 411 characters detected.
• 2 fps: 659 characters detected.
• 5 fps: 1486 characters detected. Conclusion: Using a higher sampling rate (e.g., 5fps) is critical for capturing rapid dialogue in short dramas, as 1fps misses nearly 70% of the textual content.

We generated Japanese audio samples across 12 different speakers using zero-shot voice cloning.

EdgeTTS uses a single fixed neural voice (ja-JP-NanamiNeural) without speaker cloning; per-speaker CER is not applicable.

We evaluated the ability of Qwen3-VL to translate processed Chinese subtitles into Japanese, comparing zero-shot and fine-tuned performance against several baselines.

• Fine-tuning (Rank 32, Alpha 64, up to 20 epochs with early stopping): LoRA fine-tuned Qwen3-VL-4B achieves BLEU 29.99 — a +61.8% relative gain over its zero-shot baseline.
• NLLB-200 scores only 2.91 BLEU, indicating that general-purpose MT models are poorly suited for the colloquial, speaker-tagged subtitle format used in this dataset.
• Note: Qwen2.5-7B and NLLB-200 were re-evaluated with corrected inference scripts (increased generation length, corrected sacrebleu API); earlier reported values were derived from truncated outputs.

We implemented an Adaptive Fusion strategy that combines the strengths of Whisper (ASR) and Qwen3-VL (OCR). The system dynamically adjusts its trust in OCR based on ASR confidence (avg_logprob).

• Result: F5-TTS struggled with the cross-lingual zero-shot task using short (3-5s) reference audio, resulting in higher WER compared to GPT-SoVITS.

# 1. Subtitle Recognition Environment (Python 3.8)
source venv/bin/activate

# 2. TTS Environment (Python 3.11 - recommended for best compatibility)
source venv_tts/bin/activate

# Activate venv
source venv/bin/activate

# Run Inference (Qwen2-VL)
# This processes all mp4 files in the data directory and saves results to baseline_results/
python speech2text/run_qwen3vl_inference.py

# Run Evaluation
# Calculates CER, BLEU, chrF++
python speech2text/qwen3vl_evaluator.py

# Activate venv_tts
source venv_tts/bin/activate

# 1. Preproccess Audio (Optional, extracted data already provided)
python tts/preprocess_audio.py

# 2. Run Synthesis (GPT-SoVITS)
# Generates Japanese speech for all speakers in data/tts_ref_audio/
python tts/run_tts_synthesis.py

# 3. Evaluate TTS (Whisper ASR)
# Calculates WER on generated samples
python tts/evaluate_tts.py

In response to reviewer feedback, we conducted additional ablation studies, baselines, and a system demonstration.

We assessed the impact of visual context on translation quality using Qwen3-VL.

• Method: Comparing translation BLEU scores with and without video frame input.

+3.35 BLEU improvement with visual context (20-episode subset, Qwen3-VL zero-shot).

We evaluated how the duration of the reference audio affects zero-shot speaker similarity.

• Method: Comparing similarity scores for short (3s) vs. long (10s) reference prompts using F5-TTS.

We implemented a proof-of-concept pipeline run_end_to_end.py that fully automates the workflow:

1. Video Ingestion: Reads .mp4 file.
2. Visual Extraction: Qwen3-VL extracts subtitles (OCR).
3. Translation: Qwen3-VL translates text to Japanese.
4. Audio Synthesis: F5-TTS generates dubbed audio.
5. Dubbing: FFmpeg merges audio back to video. Note: The final dubbing step (FFmpeg) currently has environmental limitations (libavutil), but the AI components function successfully.