ClothesTok

Inspiration

My mom told me about how she once decided to find a dress to buy on TikTok but TikTok keeps recommending her clothes that don't fit well on her. I figured it was probably because TikTok is only basing on the video she likes or watches for a long time, which can be people wearing clothes that look good on themselves but not necessarily for my mom. Why don't I build a model that finds only TikTok videos of people that look like her and recommend the clothes they wear to her? Introducing ClothesTok - Bao

What it does

ClothesTok leverages ML algorithms to analyze users' appearance (pose, hair, skin, etc.) and suggest personalized clothes items for them. We strive to deliver high suitability by training the model on dataset of well-received and popular Tiktok #fashion videos. We utilize SegFormer for segmenting details on every frame of the videos to model all matching characteristics between humans and clothes. Moreover, ClothesTok ships virtual try-on feature for users with state-of-the-art diffusion model. Enabling shoppers to see how clothes look on themselves will build trust in the products and dramatically enhance the shopping experience. Each item from our lists is directly sourced from Tiktok Shop, providing users with a seamless way to explore, view, and purchase products on the platform. Furthermore, ClothesTok allows users to follow leading fashion trends by interacting with those hot shopping items everyday. This feature not only keeps users up-to-date with the latest trends but also assists them heavily in making informed purchase decisions.

How we built it

Frontend: Next.js, Typescript, React, TailwindCSS, Deployed on Vercel
Backend: FastAPI, Python, Deployed on Render
ML: SegFormer, OOTDiffusion
API: HuggingFace Inference API

diagram

Challenges we ran into

As the try-on technology heavily depends on generative AI, we strive hard to eliminate hallucination and improve model performance through various strategies such as removing background and clearly segmenting body parts. Besides, we worked hard on connecting frontend and backend through API calls to provide seamless user experiences, so we are just so proud that we're done now :D

Accomplishments that we're proud of

We enjoyed rapidly developing try-on feature to better match human appearances, dramatically improving from basic AR/ 3D scanning strategies to generative AI. We are proud of collaborating with each other and working effectively despite time constraints.

What we learned

We learn about various technologies on computer vision and generative AI. Most importantly, we learn the mindset to build a dedicated product that solves our daily Tiktok scrolling problems.