Graduate Course (Master's / PhD level) at SUTD

Instructor: Assoc. Prof Dorien Herremans

This repository contains course materials for a graduate-level course on multimodal generative AI. The course covers foundational vision and audio models, multimodal alignment, retrieval-augmented generation, and agentic AI systems — combining lectures, hands-on labs, and seminar paper discussions.

Note: Prof. Wenxuan teaches the NLP part of this course, that part is not included in this repo.

#	Topic	Slides	iPython Notebook	Papers
1	Foundational Vision Models	PDF	part 1 · part 2	—
2	Foundational Audio Models	PDF		—
3	Multimodal Models & Alignment	PDF	—	5 papers
4	Retrieval-Augmented Generation (RAG)	PDF		5 papers
5	Agentic AI & Tool-Using Systems	—		—

From convolutional networks to modern vision transformers and generative models.

Topics covered:

• CNNs to Vision Transformers (ViT)
• Self-supervised visual representations: SimCLR, DINOv2
• Generative models for vision: DALL·E, Stable Diffusion
• Diffusion models: core concepts and training

Materials: Slides PDF · Slides (interactive) · Lab: part 1 · part 2

From signal representations to neural audio generation.

Topics covered:

• Spectrograms and audio tokenization
• Neural audio encoders: EnCodec, VQ-VAE
• Autoregressive models: WaveNet, Jukebox, AudioLM
• Diffusion for audio: Stable Audio

Materials: Slides PDF · Slides (interactive) · Lab:

Connecting modalities through shared representations and cross-modal learning.

Topics covered:

• Multimodal representations: CLIP, CLAP, MuLAN
• Architectures for merging modalities
• Seminar: discussion of 5 foundational papers

Materials: Slides PDF · Slides (interactive)

Grounding large generative models with external knowledge.

Topics covered:

• RAG pipeline architecture and components
• Vector databases and similarity metrics
• Grounding strategies: prompt patterns, multimodal RAG
• Hands-on lab

Materials: Slides PDF · Slides (interactive) · Lab:

Building and evaluating AI systems that reason, plan, and use tools.

Topics covered:

• Scratchpads and chain-of-thought reasoning
• Tool use and function calling
• Model Context Protocol (MCP)
• Workflow orchestration with LangGraph
• Safety evaluation with NVIDIA Garak
• Multi-agent coordination patterns

Materials: Lab:

.
├── 01-foundational-vision/          # Foundational Vision Models
│   ├── Lecture-Foundational-Vision-Models.pdf
│   └── lab_01-foundational-vision/lab_Foundational_vision_part_I.ipynb
│   └── lab_01-foundational-vision/lab_Foundational_vision_part_II.ipynb
├── 02-foundational-audio/          # Foundational Audio Models
│   ├── Lecture-Foundational-Generative-Audio-Models.pdf
│   └── lab_foundational_audio.ipynb
├── 03-multimodal-alignment/          # Multimodal Models & Alignment
│   └── Lecture-Multimodal-Alignment.pdf
├── 04-retrieval-augmented-generation/          # Retrieval-Augmented Generation
│   ├── Lecture-Grounding-and-RAG-Systems.pdf
│   └── lab_RAG.ipynb
└── 05-agentic-ai/          # Agentic AI & Tool-Using Systems
    └── lab_agentic_AI_MCP_LangGraph.ipynb

Students are expected to have:

• Familiarity with deep learning fundamentals (transformers, backpropagation)
• Python proficiency
• Basic experience with PyTorch or a similar framework

© Dorien Herremans. CC BY-NC 4.0 licence.

If you use or adapt these materials, please cite:

@misc{herremans2025multimodal,
  author    = {Dorien Herremans},
  title     = {Multimodal Generative AI: Graduate Course Materials},
  year      = {2026},
  publisher = {GitHub},
  url       = {https://github.com/dorienh/multimodal-generative-ai}
}

Questions or feedback? Open an issue or reach out via dorienherremans.com.