• ☐ ExperimentsOnSKAttentions for ablation experiments.
• ✅ 12/13/2024 Rewrite the code for the Gradio app.
• ✅ 12/12/2024 Added HelloMeme V2 (synchronize code from the ComfyUI repo).
• ✅ 11/14/2024 Added the HMControlNet2 module
• ✅ 11/12/2024 Added a newly fine-tuned version of Animatediff with a patch size of 12, which uses less VRAM (Tested on 2080Ti).
• ✅ 11/5/2024 ComfyUI interface for HelloMeme.
• ✅ 11/1/2024 Release the code for the core functionalities..

This repository contains the official code implementation of the paper HelloMeme. Any updates related to the code or models from the paper will be posted here. The code for the ablation experiments discussed in the paper will be added to the ExperimentsOnSKAttentions section. Additionally, we plan to release a ComfyUI interface for HelloMeme, with updates posted here as well.

conda create -n hellomeme python=3.10.11
conda activate hellomeme

To install the latest version of PyTorch, please refer to the official PyTorch website for detailed installation instructions. Additionally, the code will invoke the system's ffmpeg command for video and audio editing, so the runtime environment must have ffmpeg pre-installed. For installation guidance, please refer to the official FFmpeg website.

pip install diffusers transformers einops scipy opencv-python tqdm pillow onnxruntime onnx safetensors accelerate peft

git clone https://github.com/HelloVision/HelloMeme
cd HelloMeme

python inference_image.py # for image generation
python inference_video.py # for video generation

We recommend setting up the environment with conda.

pip install gradio
pip install imageio[ffmpeg]

run python app.py

After run the app, all models will be downloaded. Longer the driver video, more VRAM will need.

The input for the image generation script inference_image.py consists of a reference image and a drive image, as shown in the figure below:

Reference Image

Drive Image

The output of the image generation script is shown below:

Based on SD1.5

Based on disneyPixarCartoon

The input for the video generation script inference_video.py consists of a reference image and a drive video, as shown in the figure below:

Reference Image

Drive Video

The output of the video generation script is shown below:

Based on epicrealism

Based on disneyPixarCartoon

Our models are all hosted on 🤗, and the startup script will download them automatically. The specific model information is as follows:

model	size	url	Info
songkey/hm_reference	312M		The weights of the ReferenceAdapter module
songkey/hm_control	149M		The weights of the HMControlNet module
songkey/hm_animatediff	835M		The weights of the Turned Animatediff (patch size 16)
songkey/hm_animatediff_frame12	835M		The weights of the Turned Animatediff (patch size 12)
hello_3dmm.onnx	311M		For face RT Extractor
hello_arkit_blendshape.onnx	9.11M		Extract ARKit blendshape parameters
hello_face_det.onnx	317K		Face Detector
hello_face_landmark.onnx	2.87M		Face Landmarks (222 points)

Our pipeline also supports loading stylized base models (safetensors). For video generation tasks, using some customized models for portrait generation, such as Realistic Vision V6.0 B1, can produce better results. You can download checkpoints and loras into the directories pretrained_models/ and pretrained_models/loras/, respectively.

Thanks to 🤗 for providing diffusers, which has greatly enhanced development efficiency in diffusion-related work. We also drew considerable inspiration from MagicAnimate and EMO, and Animatediff allowed us to implement the video version at a very low cost. Finally, we thank our colleagues Shengjie Wu and Zemin An, whose foundational modules played a significant role in this work.

@misc{zhang2024hellomemeintegratingspatialknitting,
        title={HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models}, 
        author={Shengkai Zhang and Nianhong Jiao and Tian Li and Chaojie Yang and Chenhui Xue and Boya Niu and Jun Gao},
        year={2024},
        eprint={2410.22901},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2410.22901}, 
  }

Shengkai Zhang ([email protected])