A minimal Open OnDemand app for launching a Gradio-based LLM chatbot on UAB's Cheaha HPC cluster.

This was modified from and inspired by Wake Forest's HPC example of OOD-GradioApps.

• Project Background
• Install & Setup
• Usage
• Directory Structure
• Contributing
• License
• Authors

This project provides a simple, reproducible interface for deploying a local LLM chatbot using Gradio on Cheaha via Open OnDemand. It is designed for biologists and computational biologists who need interactive, on-demand access to language models for research and data exploration.

• Python version: 3.11+
• Other software: Gradio, llama-cpp-python, build tools (GCC, CMake, OpenBLAS, CUDA)

1. Create and activate a Python virtual environment:

   module load Anaconda3
   module load GCCcore/11.4.0
   module load GCC/11.4.0             # Skip if not available
   module load CUDA/11.8.0
   module load cuDNN/8.9.2.26-CUDA-11.8.0
   module load CMake/3.26.4-GCCcore-11.4.0
   
   # Create and activate conda env
   conda env create -f env-chatbot.yml
   conda activate env-chatbot
   
   # Set compiler environments
   export CC=$(which gcc)
   export CXX=$(which g++)
   export CUDAHOSTCXX=$CXX
   export CUDACXX=$(which nvcc)

2. Upgrade pip and install dependencies:

   CMAKE_ARGS="-DLLAMA_CUDA=on -DLLAMA_CUBLAS=on -DLLAMA_CUDA_FORCE_DMMV=on -DLLAMA_NATIVE=off" pip install llama-cpp-python==0.3.14 --no-binary :all:

3. Download a pre-trained LLM model (GGUF format):

   mkdir -p ${HOME}/llm
   # Download a smaller, more reliable model
   wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf -O ${HOME}/llm/Phi-3-mini-4k-instruct-q4.gguf

1. Log in to Open OnDemand.
2. Select "Gradio Chatbot" from the Interactive Apps menu.
3. Fill in the resource form (partition, CPUs, memory, etc.).
4. Submit the job and wait for the session to start.
5. Click the provided link to access the Gradio chatbot interface.

$ tree -a ood-gradio-chatbot/
ood-gradio-chatbot/
├── README.md               # This file
├── icon.png                # App icon
├── manifest.yml            # App metadata
├── form.yml.erb            # OOD form definition
├── submit.yml.erb          # OOD SLURM submission script
├── template/
│   ├── before.sh.erb       # Pre-launch environment setup
│   ├── script.sh.erb       # Launch script
│   └── after.sh.erb        # Post-launch script (optional)
├── view.html.erb           # OOD web interface button
└── ...                     # (other files as needed)

We welcome contributions! See the docs for guidelines.

View the LICENSE for this project.

• Shaurita Hutchins email | Graduate Research Assistant | sdhutchins