Inspiration
I hate doing frontend styling. It is tedious and very visual which causes most coding assistants like GPT4 to not be very helpful. I built upon the work of these UC Berkeley researchers to extend their results to frontend web development.
What it does
The model receives 3 inputs: a target image for what the final web page should look, a randomly initialized program in the custom made DSL, and the rendered result of the DSL program. Then the model suggests edits to the program based on the difference in the target image and the rendered program. The model is constrained during token generation to only suggest syntactically valid edits. This process repeats until the model achieves the target image.
The model is trained like a diffusion model on program space. Randomly sampled programs in the DSL have their abstract syntax trees randomly mutated, then the model learns to reverse this noise to recover the original program conditioned on the target image which is the rendered original program.
Once trained, the model can be given a diagram (e.g. a Figma board) of the desired web page then the model will iteratively build the front-end until the diagram is implemented. It could likely work on less-precise specs like hand drawn sketches of a web page.
How we built it
Wrote a context free programming language and a transpiler with Lark to convert DSL programs to HTML+CSS. I began training the model in Google Colab.
Challenges we ran into
The DSL had to be structured a very particular way so that it could be randomly sampled and mutated. This made it more difficult to add functionality to the DSL because it also had to work with the sampler and mutator.
Also the model trains very slowly. The main bottleneck is the dataloading process. Currently the DSL is rendered by making it into a PDF then screenshotting the PDF. This can definitely be optimized.
What's next for StableDOM
Finish training the model. Expand the DSL. Scale compute to expand model's proficiency to the rest of HTML+CSS and frontend frameworks like React.
Log in or sign up for Devpost to join the conversation.