Meshverse

Meshverse

Inspiration

Our inspiration for this idea stemmed from observing one of Speckle's primary goals, which is facilitating the seamless exchange of intricate geometric and attribute data among diverse software platforms like Revit, AutoCAD, Blender, and more. This mission aims to simplify data creation and management within the AEC industry by allowing users to effectively use all of the tools at their disposal without having to deal with the overhead of transferring progress and data throughout a plethora of different individual applications.

What it does

Our project allows users to either to upload a file from their computer or use their camera to create a 3D mesh using stable diffusion. The user can then download this mesh, add it as a stream into Speckle's client, or receive Speckle's data on the stream that they created. Some other features that we integrated includes google sign in via next-auth, a MongoDB database that stores image urls and user info data, the ability to view the gallary ( a collection of all other meshes generated by different users), and the ability to see what 3d meshes a specific user has generated.

How we built it

Our backend was built off of a python Flask server. Through this server we were able to connect to a MongoDB database, utilize Speckle's Python SDK to connect to Speckle's client, created multiple CRUD operations for the users and image urls, and uses Gradio's api to generate the AI 3D Meshes. On the frontend side, we utilized Next.js and Tailwind to create visually appealing and responsive pages, seamless navigation, and connectiveness to our backend. We also used Framer Motion for smooth transitions and animations, and lastly for the frontend we used Figma to create beautiful neon background glows and to edit pictures.

Challenges we ran into

One challenge we ran into was figuring out how to store images on the backend, and communicate these images onto the frontend. It is not just as simple as passing an image object between frontend and backend, and required several more tedious well-connected steps. Another challenge we ran into was converting the .obj files from Python to Speckle. In Python the way we read in .obj files is different to how Speckle reads them, so we had to manipulate them in order to send meshes directly into Speckle's Client through our backend.

Accomplishments that we're proud of

We are proud that we managed to fininsh our web application without any glaring errors or deficienties, and we hope that our application can lead to or inspire positive change to the AEC industry as a whole.

What we learned

This hackathon was our first time interacting with Speckle, so we learned how to connect to Speckle's Client via Speckle's Python SDK, send 3D meshes directly into Speckle's client automatically, receive data from a stream's id in Speckle with a click of a button. Additionally, we learned more about stable diffusion models that generate 3D Meshes and how to incorperate them seamlessy into our web application.

What's next for Meshverse

There are many things that we can do with Meshverse in the future. This include adding more sign in authentication methods, adding multiple security layers via middleware and JWT secret tokens, hosting the website to make it availible to a broader range of users, adding additional error handling on the frontend, tracking more data in our MongoDB database, code optimization, and code refactoring. Overall, Meshverse is only a fraction of what we want it to be, and we look forward to further developing it in the future.