DAWZY

DAWZY is a new paradigm for music creation. Whether you speak, hum, or type, DAWZY works alongside you in any digital audio workstation to help bring your musical ideas to life.

Comment

Inspiration🎧

Music production software is incredibly powerful, but often intimidating. Many aspiring creators hear melodies in their heads but lack the technical knowledge or time to bring those ideas to life. We wanted to change that. Our goal was to build an assistant that empowers anyone, beginners & seasoned artists, to turn their musical ideas into reality with just their voice or a simple prompt.

What it does 🤖

DAWZY is an AI music production assistant that integrates with any DAW (digital audio workstation). It supports:

  • Voice-to-Voice Communication: Have a natural conversation with the assistant about your music project.
  • Text-to-Music Prompting: Describe what you want and let DAWZY generate instruments, tracks, and effects.
  • Humming-to-MIDI Conversion: Hum a melody and have it transcribed into a playable MIDI part.
  • DAW Control via Language: Tell DAWZY to add reverb, adjust EQs, or create new instruments, and it writes scripts to control your DAW environment directly.
  • Contextual Memory: DAWZY keeps track of your session and helps build on ideas progressively.

How we built it 🛠

  • Frontend: A React-based chat interface with support for audio messages, speech input, and real-time feedback.
  • Backend: A Flask server acting as a bridge between user input, language models, and Reaper (our current DAW prototype).
  • LLMs: OpenAI models power the conversational and action logic, while Claude handles DAW scripting.
  • Music Generation: We integrated Beatoven.ai for prompt-based music generation.
  • Speech-to-Speech: We designed the architecture to allow natural voice dialogue with OpenAI's new speech model.
  • Audio Conversion Tools: We implemented humming-to-MIDI by recording audio, converting it to WAV, and sending it to our MIDI processing logic.
  • Reaper Integration: Using python-reapy, we remotely manipulate Reaper sessions from user commands.

Challenges we ran into⚠️

  • Audio Handling: Getting consistent recording formats and playback behavior across browsers was surprisingly complex.

  • LLM Integration: Coordinating different models (chat, voice, and action) while maintaining session continuity took careful planning.

  • DAW Control: Reaper exposes a lot of functionality, but translating natural language into precise DAW scripting was a major challenge.

  • Latency and Feedback: Ensuring that the assistant felt responsive, even when handling complex tasks like MIDI conversion, required efficient message handling and is something we wish to improve on in the future

Accomplishments that we're proud of 🏆

  • A Working End-to-End Flow: From voice input to MIDI playback, we created a seamless creative loop.

  • Intelligent DAW Manipulation: DAWZY can write and run scripts to make edits inside a live DAW session.

  • Multimodal Interaction: Users can interact by typing, speaking, or even humming.

  • User-Friendly UI: The chat interface feels intuitive and responsive even when performing behind-the-scenes audio and AI processing.

What we learned 📚

  • How to integrate multiple AI services together in real time.

  • The importance of UX when building tools for creative professionals.

  • Audio processing and format compatibility challenges across platforms.

  • Designing systems that adapt LLMs to real-world control tasks beyond text generation.

What's next for DAWZY 🚀

  • Full DAW Support: We plan to expand DAWZY beyond Reaper to support other popular DAWs like Ableton, Logic Pro, and FL Studio.

  • Real-Time MIDI Feedback: Users will be able to edit melodies or chord progressions conversationally while hearing live previews.

  • On-Device Voice Looping and Jamming: Think of DAWZY as your AI bandmate.

  • Plugin Suggestion and Effect Chain Design: DAWZY will recommend and auto-configure effect chains for genres or moods.

  • Deeper VAPI Integration: Fully enable two-way natural voice conversations.

Our long-term vision is to make DAWZY the ultimate collaborator for music producers—creative, technical, and always ready to jam.

Built With

Share this project:

Updates