🏆 FLUX.1 Kontext [dev] Hackathon Submission - InstructDesign Flow on Devpost

Transform web interfaces iteratively with natural language instructions

Transform web interfaces with natural language instructions using our fine-tuned FLUX.1 Kontext model. This project enables AI-driven web design transformations through simple text prompts, trained on a curated dataset of 900+ webpage transformation pairs.

InstructDesign Flow is a specialized fine-tuning of FLUX.1 Kontext [dev] that understands and executes web design transformations through natural language. Simply provide a webpage screenshot and describe the desired changes - the model handles the rest.

• 100+ Pre-defined Design Presets: From dark mode to cyberpunk themes
• Natural Language Control: Describe changes in plain English
• Consistent Transformations: 85%+ instruction adherence rate
• End-to-End Pipeline Development for Production: Dockerized with automatic model downloads

• Dataset Repository: HuggingFace - instructdesign-kontext
• Size: 937 webpage transformation pairs
• Sample Available: 100 representative pairs publicly available
• Format: Original screenshots + transformed outputs + text instructions
• Coverage: UI/UX redesigns, theme changes, layout modifications, style transfers

• Training Framework: ostris/ai-toolkit
• Configuration: flux_kontext_training_v4_consolidated.yaml
• Development Time: 7 days
• Hardware: NVIDIA H100 GPU with 80GB VRAM
• Training Steps: 16,000
• Checkpoints Saved: Every 1000 steps

Model Architecture:
  Base Model: black-forest-labs/FLUX.1-Kontext-dev
  Type: LoRA (Low-Rank Adaptation)
  Rank: 256
  Alpha: 256

Training Parameters:
  Batch Size: 2
  Learning Rate: 7e-5
  Optimizer: AdamW 8-bit
  Gradient Accumulation: 1
  Noise Scheduler: FlowMatch
  Timestep Type: Sigmoid
  Resolution: [512, 768, 1024]

Dataset Settings:
  Caption Dropout: 5%
  Cache Latents: True
  Control Images: _original.jpg suffix
  Output Images: _output.jpg suffix

• Model Repository: HuggingFace - instructdesign-kontext
• LoRA Weights: flux_kontext_lora_v4_consolidated_000010000.safetensors
• Base Model: flux1-kontext-dev.safetensors
• Text Encoders: CLIP-L + T5-XXL (FP8)
• VAE: ae.safetensors

• NVIDIA GPU (16GB+ VRAM recommended)
• Docker & Docker Compose
• NVIDIA Container Toolkit
• ~100GB disk space (for models and Docker images)

1. Clone the repository:

git clone https://github.com/tercumantanumut/InstructDesign-Flow
cd InstructDesign-Flow

2. Start the services (models download automatically):

docker-compose up -d

The services will be available at:

• API: http://localhost:8000
• ComfyUI: http://localhost:8188
• Frontend: http://localhost:3000 (if running)

1. Theme Changes: Dark mode, light mode, high contrast
2. Design Systems: Material Design, iOS, Windows Metro
3. Style Effects: Glassmorphism, neumorphism, brutalism
4. Layout Modifications: Mobile-first, dashboard, e-commerce
5. Industry Specific: SaaS, portfolio, blog, landing page
6. Creative Themes: Cyberpunk, retro, minimalist, maximalist
7. Device Mockups: iPhone, MacBook, billboard placements
8. Accessibility: High contrast, larger fonts, better spacing

curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "positive_prompt": "Transform this interface to dark mode with modern aesthetics",
    "input_image": "inputs/sample_webpage.png",
    "seed": 42
  }'

curl -X POST http://localhost:8000/api/generate \
  -d '{
    "positive_prompt": "Apply glassmorphism_ui aesthetic",
    "input_image": "https://example.com/screenshot.png",
    "return_base64": true
  }'

import requests
import json

presets = ["dark_mode", "mobile_responsive", "minimalist_clean"]
for preset in presets:
    response = requests.post("http://localhost:8000/api/generate",
        json={
            "positive_prompt": f"Apply {preset} transformation",
            "input_image": "test.png",
            "seed": 42
        })
    print(f"{preset}: {response.json()}")

InstructDesign-Flow/
├── docker-compose.yml         # Service orchestration
├── build/
│   └── Dockerfile            # ComfyUI + models container
├── workflow_api.json         # ComfyUI workflow definition
├── inputs/                   # Input images directory
├── output/                   # Generated outputs
├── deeployd-comfy/          # DeepFloyd-Comfy API wrapper (my own deployment framework)
├── comfyui-deploy-next-example/ # Frontend application
└── training/
    └── config.yaml          # Training configuration

1. comfyui-instructdesign: GPU-accelerated inference server

   • Automatic model downloads (~12GB)
   • ComfyUI with custom nodes
   • CUDA 12.8 optimized

2. instructdesign-api: FastAPI wrapper

   • Queue management
   • WebSocket support
   • Priority task processing

• Base Model: FLUX.1 Kontext [dev]
• Fine-tuning Method: LoRA with rank 256
• Training Duration: 10,000 steps
• Inference Time: 45-70 seconds per 1024x1024 image
• VRAM Usage: ~14GB during inference
• Consistency: 85%+ instruction following

• Queue Capacity: 1000 concurrent tasks
• Worker Threads: Auto-scaling 1-4
• Average Latency: 55 seconds
• Success Rate: 95%+
• Supported Formats: PNG, JPEG, WebP

"Transform this webpage to dark mode with purple accents"
"Apply material design 3 principles with rounded corners"
"Convert to cyberpunk aesthetic with neon colors"
"Make this mobile-responsive with touch-friendly buttons"
"Add glassmorphism effects to all card elements"

"Redesign this interface for accessibility, increasing contrast
 and font sizes while maintaining the original brand colors"

"Transform this desktop layout to a mobile-first design with
 hamburger navigation and thumb-friendly interaction zones"

"Apply a premium SaaS aesthetic with gradient backgrounds,
 subtle shadows, and modern typography"

1. Prepare your dataset:

   • Collect webpage screenshots (original)
   • Create transformed versions (output)
   • Write transformation descriptions (captions)

2. Configure training:

git clone https://github.com/ostris/ai-toolkit
cd ai-toolkit
# Copy our training config
cp /path/to/flux_kontext_training_v4_consolidated.yaml config/

3. Start training:

python run.py config/flux_kontext_training_v4_consolidated.yaml

# TensorBoard logs
tensorboard --logdir=output/flux_kontext_lora_v4_consolidated/logs

# Sample outputs every 1000 steps
ls output/flux_kontext_lora_v4_consolidated/samples/

• Competition: FLUX.1 Kontext Hackathon
• Model Weights: HuggingFace Model Hub
• Dataset Samples: HuggingFace Datasets
• Training Framework: AI Toolkit by Ostris
• Base Model: FLUX.1 Kontext [dev]
• API Framework: DeepFloyd-Comfy (My ComfyUI deployment solution)
• Frontend Foundation: ComfyUI Deploy Next.js Example

This project was developed for the Black Forest Labs FLUX.1 Kontext [dev] Hackathon:

• Development Period: 7 days
• Team: Umut Tan
• Focus: Web interface transformation through natural language
• Innovation: First LoRA fine-tune specifically for web design transformations

We welcome contributions! Areas for improvement:

• Additional transformation presets
• Frontend UI enhancements
• Performance optimizations
• Dataset expansion
• Documentation improvements

This project uses the FLUX.1 Kontext [dev] model. Please refer to:

• FLUX.1 Kontext License
• Dataset: Research and educational use
• Code: MIT License

• Black Forest Labs for FLUX.1 Kontext [dev] model
• Comfy.org - ecosystem support
• Ostris for the excellent AI Toolkit training framework
• ComfyUI community for the powerful inference infrastructure
• ComfyDeploy for the Next.js frontend example that served as the foundation for our UI

• fal.ai - Hackathon co-sponsor and compute infrastructure
• Black Forest Labs - Model provider and hackathon host
• NVIDIA - Hackathon partner

• HuggingFace for model and dataset hosting
• DeepFloyd-Comfy - My own ComfyUI deployment application used for the API wrapper

• The entire FLUX.1 and ComfyUI community for their support and feedback
• All hackathon participants for pushing the boundaries of AI-driven design

• Complete model training (16,000 steps)
• Docker containerization with auto-download
• API implementation with queue management
• Dataset publication on HuggingFace
• Frontend with 100+ presets
• Web-based playground interface
• Fine-tuning guide and tutorials

📧 Contact: [email protected] | 🏆 Hackathon: FLUX.1 Kontext [dev]