LumaChords, computer vision powered, for turning piano tutorial videos into MIDI, MEI and musical notation videos

Convert piano tutorial videos into MIDI, MEI files or videos with beautiful sheet music overlays! LumaChords uses computer vision to detect piano keys and falling notes in Synthesia-style tutorial videos, then generates musical notation synchronized with video playback.

Great for piano learners, enthusiasts, hobbyists, and music educators who want to extract note information from tutorial or performance videos and view it as sheet music.

It is also a strong end-to-end sample project for computer vision enthusiasts, with a multi-stage pipeline that includes specialized strategies for challenging cases, plus both a game-like GUI built with Pygame and OpenGL and a headless terminal interface.

LumaChords was developed primarily for personal use, educational purposes, and experimentation, combining software engineering, computer vision, and musical interests rather than commercial goals. For the same reason, it has been open-sourced.

LumaChords was designed and developed by a human with AI-assistance, is is not a vibecoded project. The initial commit history is intentionally clean, since the earlier draft repository had many (~250) experimental commits.

As a piano learner and enthusiast, I often explore different sources for my favorite songs I can play, including sheet music, piano tutorial videos, and MIDI files, both free and paid. However, these sources often have significant differences in musical interpretation, such as simplified chords, arpeggios, and other arrangements, and many of them do not sound close enough to the original song.

Sometimes I found piano tutorial videos that matched what I was looking for, but they often had no corresponding sheet music available, whether free or paid.

There are open-source projects that cover certain parts of LumaChords’ overall goal, but they may focus on only a narrow subset of tutorial video styles or rely heavily on manual input and setup, such as defining keybed and note regions, specifying video-specific colors, or manual calibration by the user. In contrast, LumaChords is designed to detect and extract the necessary information on its own, without requiring additional manual input or calibration.

So, as an engineer, I decided to combine my technical and musical skills together, then this project was born.

At the technical side, a deep learning segmentation model could have been used, but I intentionally put myself into a challenge with the following requirements and constraints:

• The end-to-end application must have both a GUI and a headless terminal user interface,
• The GUI must provide two modes: a basic mode that minimizes technical detail and lets the user focus on the outcome, and an advanced mode that exposes the detection process from multiple perspectives,
• Have multiple video input backends: FFmpeg and OpenCV. OpenCV is used when FFmpeg is not located on the system as a fallback,
• Have multiple note detection strategies: one for videos with textured backgrounds, and another for videos with sparse or near-monotonic backgrounds,
• Have multiple hands detection strategies: one for videos with visible human hands, and another for videos with colored keys,
• Use only rule-based classical computer vision techniques, ideally no machine learning methods at all (only K-Means Clustering is used in FFT Analysis),
• Avoid any GPU requirement for computation, except for video file processing and GUI rendering,
• Just rely on Numpy's vectorized calculation operations and CPU SIMD capabilities wherever possible,
• Provide (as much as) smooth, low-latency and responsive GUI using OpenGL.

For technical details of the multi-stage pipeline of this project, you can check out docs/METHODOLOGY.md file.

This project contains mutliple styles of installation methods available.

FFmpeg is a third-party multimedia framework that LumaChords optionally uses to process multimedia files. FFmpeg is not included and distributed with LumaChords. LumaChords does not distribute FFmpeg binaries. Users (you) are responsible for installing FFmpeg independently according to their requirements.

LumaChords can operate using included OpenCV video backend, but having FFmpeg is recommended. You can follow instructions at Installing FFmpeg document for more information.

If you want to install LumaChords as a Python package;

(Recommended)

You can install it in a virtual environment as follows:

• Ensure uv tool is installed. If not, check out Astral's documentation at Installing uv,

• Execute the following:

$ uv tool install lumachords --python 3.12
$ lumachords # to run the application

(Not recommended)

Or, you can install it directly calling:

$ pip install lumachords
$ lumachords # to run the application

• Ensure uv tool is installed. If not, check out Astral's documentation at Installing uv,

• Clone this repo and enter into this directory,

On Linux/MacOS:

$ cd lumachords
$ uv venv
$ source .venv/bin/activate
$ uv sync
$ uv run lumachords

On Microsoft Windows:

$ cd lumachords
$ uv venv
$ .venv\Scripts\activate
$ uv sync
$ uv run lumachords

If you prefer using prebuilt executables, you can download them from the project's GitHub Releases page.

• Open the latest release and check the Assets section.
• Download the archive that matches your operating system and CPU architecture.
• Extract the downloaded archive to a directory of your choice.
• Run the included executable file from the extracted directory.

The published executables are currently not code-signed. Depending on your operating system and security configuration, they may be blocked, require additional confirmation, or may not run without manual allowance. If that happens, use one of the other available installation methods.

Release artifacts are platform-specific, and some platforms may not be available for every version. For current platform-related limitations, see KNOWN-ISSUES.md.

• Clone this repo and enter into this directory,
• Run the following script and check out the "dist" directory in the project directory. It will generate executable files only for the OS/platform on your machine.

$ cd lumachords
$ ./scripts/build.sh

• For VSCode and its counterparts, this repository contains debug mode launching definitions in .vscode/launch.json file.

• Ensure uv tool is installed. If not, check out Astral's documentation at Installing uv,

• Clone this repo and enter into this directory,

On Linux/MacOS:

$ cd lumachords
$ uv venv
$ source .venv/bin/activate
$ uv sync

On Microsoft Windows:

$ cd lumachords
$ uv venv
$ .venv\Scripts\activate
$ uv sync

• Start debugging in your IDE.

• This repository contains experiment.ipynb and continuity_experiment.ipynb Python notebooks containing experiments on one frame image or series of frame images at Python notebook environment. While development of this project, the Python Notebook extension of VSCode is used.
• The algorithms in the whole pipeline were developed and tested using these notebooks.
• You can find experiment_samples/ and tests/data/ directories containing sample frames to be used in experiments, in categorized form.
• The continuity subfolders of them contain consecutive frame series to experiment on time sequence.

This project does not claim to provide fully successful or fully accurate results, and no such guarantee is given. While development, LumaChords was tested on various styles and formats of piano tutorial videos. Its multi-stage pipeline includes specialized strategies to handle many of these differences, but it is normal to output weird outputs or no output for some video inputs. For example, LumaChords won’t be able to detect notes correctly in videos that already contain sheet music overlays or fixed text at the top of the video.

For details of current known problems, limitations, and workarounds, see KNOWN-ISSUES.md.

The current headlines are:

• Detection Failures and Unexpected Outputs
• Staccato detection is missing
• Auto Timing is experimental
• MacOS Desktop: Latency while the first run
• Linux Desktop: Exit Hang
• Linux and MacOS: File Dialog Focus
• Windows: Save Dialog Filter Limitation
• Windows ARM64: Release Availability

You are welcome to create issues to report any bugs or problems you encounter.

If you liked and found my project helpful and valuable, I would greatly appreciate it if you could give the repo a star ⭐ on GitHub. Your support and feedback not only help the project improve and grow but also contribute to reaching a wider audience within the community. Additionally, it motivates me to create even more innovative projects in the future.

I want to thank to contributors of the awesome libraries and tools which LumaChords depends on:

These dependencies are used for development and building only. They are not included in distributed artifacts.

The input sample images are captured/extracted from third-party YouTube videos for limited research/testing use purposes only. See experiment_samples/EXPERIMENT_SOURCES.txt and tests/data/TEST_SOURCES.txt for ownership and limited-use notes.

LumaChords is licensed under the Apache License, Version 2.0. See LICENSE for the full license text, also see THIRD_PARTY_LICENSES.txt for third-party licenses.