Inspiration
I wanted the benefits of handwriting notes, while still having the maintainability of typed and easily searchable notes. I didn't want to switch from my usual app, Obsidian. I noticed that my old Windows tablet's on-screen keyboard had a handwriting option, and didn't see it on macOS, so I decided to make one.
What it does
Converts handwriting to text via a machine learning model I trained, and allows the user to copy the text.
How I built it
I used the Tauri framework, which is an alternative to Electron. Both frameworks allow programmers to make desktop apps with frontends similar to websites. However, Tauri apps are much lighter than Electron apps because they're reliant on the device's native webview rather than packing a Chromium browser. I also used Python to train my ML model. I used Jupyter Notebook to train some data and save it into a file, and then I made a separate Python script that would take in the path to the model file and the image (as bytecode) as command line arguments and output the prediction. The app uses SvelteJS as a frontend. The user draws on an HTML canvas. The canvas is duplicated behind the scenes, and a white background is added. It is then converted into PNG bytecode, which is sent into the backend (Rust), after which Rusts sends the bytecode to the Python script and returns the result.
Challenges I ran into
The bundle when deploying the app didn't work because it didn't embed the Python script and the model into the bundle. When I ran the build command on my laptop, it worked, but when GitHub Actions prepared the bundles, it didn't do so correctly. My ML model took 128x32 images (since that was ideal for the dataset I had) but 128x32 was too small for my HTML canvas, so I made it bigger. However, the image generated by the frontend is downgraded significantly when being resized. Additionally, I had trouble figuring out how to pass the bytecode into the Python script. Furthermore, style with CSS was a pain, since I didn't have as much experience with it.
Accomplishments that we're proud of
Starting something and making this much progress this quickly.
What we learned
Sometimes I need a little competition to get my projects up and running.
What's next for Violet
Making it so that it can be focused like an on-screen keyboard. Spending more time training an ML model to be better. Optimizing the model for larger images.
Log in or sign up for Devpost to join the conversation.