Since Part 1, we’ve established that we want to be modern AI engineers on a linux box which we’ve turned into a headless cluster able to serve any virtualized needs we can think up.

There is no way to navigate this potentially complex environment without going back to square one methodologically (i.e. entering commands for every little thing). It will require you accept some things and adapt to others and your mileage may vary.

1. The future of computing shouldn’t typically be limited by the interfaces of today.

2. AI research is costly in time before it’s particularly costly in terms of money. Though one can lead to another, time is what we’re looking to reinvest.

3. The pace of progress is due to work ethic among many people, all around the world, in a field defined by the bleeding edge. Operating systems of today are not built for this.

Agree? 🫣 Then you’re still reading the right series! 🤝

Let’s get back to building the future, with it in mind that these issues apply to every person hoping to build something with AI at a pace which dignifies their creativity and likely technical aptitude as well.

🐂 Rancher Control Plane UI

First, we need to install helm. It doesn’t get its own section because it’s not a concept on it’s own which needs explaining, the Kubernetes manifest packages helm calls charts is what you will fall asleep at your workstation over.

Let’s grab it, then install Rancher’s few dependencies.

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

$ helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

Create a namespace in your local cluster separate from system manifests:

$ k3s kubectl create ns cattle-system

Install cert-manager, which makes TLS/HTTPS possible, then install.

$ k3s kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.crds.yaml
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace

Finally, it’s time to install Rancher.

$ helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=YOUR_SSLIP_HOSTNAME \
  --set bootstrapPassword=admin

YOUR_SSLIP_HOSTNAME needs to be replace with the following:

<your hosts local IP (not your external IP)>.sslip.io

unless you are hoping to host this externally, in which case you absolutely can by instead entering the domain you hope you remotely connect to your host with via your own networking approach. Just be sure then to change the password to something substantially more secure.

Now, check Rancher’s deployment progress (and probably wait a bit).

$ k3s kubectl -n cattle-system rollout status deploy/rancher

When the table which prints out suggests the rancher- workload in particular is “Running”, you should see your new control plane interface at the hostname you gave it from a different machine on the local network.

Once you’ve logged in, you’ll be greeted with a less customized version of this:

Congrats! This is an enviable development cluster UX already, but it needs workload context.

💪 GPU Workloads in Kubernetes

Kubernetes uses manifests made of YAML (Yaml Ain’t Markup Language)1.

For this reason, we must first install the Kubernetes ResourceDefinitions and ResourceClasses for CUDA-enabled workloads which require driver-level GPU access2.

As I said before, we no longer need the CLI.

1️⃣ For the first repository, enter these details:

name: nvdp

url: https://nvidia.github.io/k8s-device-plugin

2️⃣ And for the second, enter these:

name: nvgfd

url: https://nvidia.github.io/gpu-feature-discovery

Once you’re done adding the repositories, you will be able to find them in the Chart search.

Go ahead and install both of these with the registered defaults.

Hard to tell what they’ve done when done. Here are the two namespaces the Nvidia helm charts made.

Now that it’s possible to talk to our GPUs through the Kubernetes cluster control plane: it’s time to use them, and in a way we are familiar with.

Back to Jupyter Notebooks (nothing is lost in our goal to engineer a better system, it’s just better)!

📝 One to Jupyterhub

We’re going to play on these words: Zero to Jupyterhub because well, we’ve skipped ahead a bit.

Similar to how you added the Nvidia repositories, let’s configure the jupyterhub one.

name: jupyterhub

url: https://hub.jupyter.org/helm-chart/

And just like before, go ahead and install — WAIT. Don’t install just yet.

Make sure you have a chance to do two things suited to your requirements:

1. create a namespace.

2. edit your JupyterHub values.yaml.

Creating a namespace is easy. I’ll let you do that.

Your JupyterHub configuration is altered significantly in two different YAML keys.

First thing is to use the config block under hub to set up authentication. Easy enough, just have faith Kubernetes will know what to do with regard to the application’s image.

hub:
  ...
  config:
    JupyterHub:
      admin_access: true
      authenticator_class: dummy
    Authenticator:
      admin_users:
        - <your account name>
    DummyAuthenticator:
      password: <desired password>

Next is the image which is pulled then used as your JupyterLab environment. As you may know, we require CUDA be installed anywhere torch and the like will be loaded. We’re going to use a pre-built image that contains that work already, and extends it upon the Jupyter-single-user image that comes in the helm chart by default.

So in essence, just copying the official Jupyter image with CUDA pre-installed.

singleuser:
  ...
  image:
    name: cogstacksystems/jupyter-singleuser-gpu
    pullPolicy: IfNotPresent
    pullSecrets: []
    tag: latest

There! When it installs, it will use this image to spin up new JupyterLab environments.

One last thing, we’re going to make sure the proxy-public service the operator creates does not clash with the Rancher UI. This should be done through the UI.

Navigate to Service Discovery → Services → scroll down → click proxy-public → and swing over to the dropdown, click Edit Config where you will change the listening port to… something else. We’re going to use 8888.

You can now use the hostname/IP of the cluster to reach JupyterLab on this port.

In a production environment, this would be handled by an ingress with a custom domain configuration such that you could deploy infinite services with associated subdomains.

Milvus

We’re going to use a Kubernetes operator to construct our Milvus memory “subsystem”3 which contains our environment’s virtual context.

I will need to beg an apology from the reader, you must return to the command line once more. But do so through Rancher nonetheless and enter in this command.

$ helm install milvus-operator \
  -n milvus-operator --create-namespace \
  --wait --wait-for-jobs \
  https://github.com/zilliztech/milvus-operator/releases/download/v0.8.4/milvus-operator-0.8.4.tgz

Check that the operator is installing accordingly:

$ kubectl get pods -n milvus-operator

While the operator is creating all of its necessary resources, one of them will be a CRD (custom resource definition) for Milvus clusters. This should take just a moment. Now, creating a new cluster to store information is as easy as saving the following as *.yml:

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: llm-mem # feel free to edit this
  labels:
    app: milvus
spec:
  mode: cluster
  dependencies: {}
  components: {}
  config: {}

with kubectl the same way we do the others:

$ kubectl apply -f llm_mem.yml

in order to validate that, we use the CRD in our kubectl command.

$ kubectl get milvus && kubectl get pods

NAME MODE STATUS UPDATED AGE

llm-mem cluster Healthy True 6m13s

…

Excellent! From there you should be able to see your milvus cluster booting up. This may also take a moment.

This diagram4 is particularly oversimplified. Those “parse” arrows are doing a lot of heavy lifting here. Thankfully, we have a short answer in the form of hygiene, our tool to convert API responses and payloads into compressed natural language structures (bullet pointed lists and the like, while saving ~15% tokens on average).

We also don’t have a traditional computing interface to parse any of this information in a contained, secure way that is transparent and easy to debug.

Instead, we have a clustered computing environment capable of that.

In Part 3, we will build a custom inference API for designing your own way around problems with data with LLMs⚡️

The LLM space moves fast. As of writing this, most of the good, viral research is about one of two things:

1. Emulating aspects of OpenAI’s ChatGPT architecture/performance

2. More efficient model paradigms

Increasingly, peers I’ve spoken to place an emphasis on the sheer amount of testing and toying around with different solutions that is required. Not just A/B’ing models but the ability to control the underlying resources, fine-tuning from several versions of datasets at once, library version control, quantization comparison, the list goes on but it includes modularizing all of this. This doesn’t even address the time required to read the aforementioned research.

As a data scientist-turned-engineer, my role is characterized by a focus on availability, monitoring, and observability in the most efficient and performant way(s). People in more sales-heavy parts of the tech industry refer to this as “AIOps” (as “ML” becomes less used every day). If we dig a little deeper into the patterns we’re able to see culturally, it’s very likely Karpathy is onto something when he suggests LLMs may be a new computing paradigm on the kernel-level (as he usually is). Let’s adopt this philosophy and think of what we are building as the bare minimum for quality research. This architecture will define a fluid LLM-based computing platform which you can use to create, host, understand, interpret any down-stream use case given what we understand about LLMs today (including the momentum of change in their research).

In this article, we’re going to look over what about implementing large language models is so time-consuming in early stages, and how to tick the important boxes we’d usually reserve for production deployments in a way that meets the pace of the experimentation phase.

In short, it’s time to move away from doing everything in a Jupyter Notebook 😬 Sorry, but it’s slowing you down!

⚗️Toolkit

To do that, we’re building a production-grade environment and cutting some corners to make the silhouette of our deployment overlap just our initial needs.

We’re getting help from several important architectural & design choices.

• kubernetes

  • clustering configuration engine for infrastructure

  • and we’ll use k3s, a lightweight version of Kubernetes that preserves production concepts

• nvidia-container-toolkit

  • GPU-passthrough for docker images

• vLLM

  • the recent ability to use quantization flags in prebuilt images with CUDA 12.x installed on an Ubuntu host makes this very straight-forward

• milvus

  • OSS, performant, scalable vector DB with great team(s) behind the product

• magnet & hygiene

  • tools we made at Prismadic to help with LLM-specific tasks & routines

• haystack

  • similar to langchain, but includes well thought-out design patterns for scalability.

• CUDA 8-compatible GPUs

  • The T4 found on AWS g4dn instances, as an example, works only up to CUDA 7.5. g5.xlarge is the most affordable type of instance that has GPUs capable of performing modern LLM research tasks available on AWS in regular quantities.

These pieces put together create an extremely configurable and adaptable environment to test n models for most downstream tasks quickly.

⚙️ Configuring your host for virtualizing GPU workloads

Assuming you have CUDA enabled on an Ubuntu 22 machine, let’s install some necessary components you might not have already as an ML researcher so that we can begin turning this computer into a zombie that exists just to serve LLMs.

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
$ sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io -y
$ sudo usermod -aG docker $USER

there, we’re just getting container and cluster virtualization tools then adding ourselves to the docker group, so moving on!

Next is to install nvidia-container-toolkit

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
$ sudo systemctl restart docker

(it’s always handy to do a quick nvidia-smi at times like this to make sure your drivers and CUDA libs are working as expected)

when this is finished installing, you’ll be using a slightly modified containerd runtime in docker which allows you to take advantage of host GPU inside the docker image context. You can think of it as what hardware passthrough is to bare-metal VM environments.

🧪 Test your GPU with vLLM

Minor changes to existing tools go a long way to pave a path of least resistance.

vLLM has good documentation for creating your own image but this requires that your CUDA version matches that of the PyTorch used in the image to build the vllm package from pip — which can present deployment hurdles on hosts where an existing CUDA installation exists which other development might already depend on. conda addresses some of this, but it’s not great for clusters like Kubernetes which puts us back to square one.

Instead of using their image, we’re going to grab a pre-built one we’ve already contributed to prior to this article as to prepare for the quantization requirement of our deployment.

Since this is already addressed, it’s as simple as:

$ sudo docker run -d -p 8080:8080 --gpus=all -e MODEL=TheBloke/Mistral-7B-Instruct-v0.1-AWQ -e QUANTIZATION=awq -e DTYPE=half ghcr.io/substratusai/vllm

docker will then:

• open 8080 to the application

• use all of your GPUs which are discoverable by CUDA

  • distributed inference!

• assign to the server a quantized model from TheBloke on huggingface

• use the new flags for the image to appropriate a non-standard model

(you may find some of this useful if you’d like to build a vllm image yourself instead, but there’s really no advantage to this as you’ll find out later when we deploy the python API)

You can delete this container now with sudo docker container stop <id> now as we’re going to deploy this as a pod with a Kubernetes manifest, which will enable reliable availability with little effort. The image is useful to k8s too though, so don’t remove it once built.

☸️ Kubernetes

Kubernetes eludes the research space and it’s no wonder why: it’s a lot of difficult abstraction. This article doesn’t beg the attention between building and philosophy though. Besides, for some reason it garners pretty intense reactions (and I’m not totally sure why…)

In any case, let’s install k3s & kubectl so that we aren’t left behind.

$ curl -sfL https://get.k3s.io | sh -
$ curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
$ chmod +x kubectl
$ sudo mv kubectl /usr/local/bin/

You should see a healthy Kubernetes cluster info dump.

Next, to quickly deploy a quantized version of the mistral-7b-instruct model in your new Kubernetes cluster, create a file called vllm.yaml and in that file, paste the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  labels:
    app: vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: ghcr.io/substratusai/vllm:latest
        ports:
        - containerPort: 8080
        env:
        - name: MODEL
          value: "TheBloke/Mistral-7B-Instruct-v0.1-AWQ"
        - name: QUANTIZATION
          value: "awq"
        - name: DTYPE
          value: "half"
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        readinessProbe:
          httpGet:
            path: /docs
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          limits:
            nvidia.com/gpu: "1" 
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 1Gi

note: we put “1” as the number of GPU in resources.limits.nvidia.com/gpu because each model will be designated to the VRAM we have allotted (and 14GB is required for the model without inference or embeddings from a vector DB in the memory!)

now, it’s as simple as:

$ k3s kubectl apply -f vllm.yaml

🎉 You should now have a highly scalable, virtualized, containerized deployment for deploying not just quantized models, but almost any model on dedicated GPU compute (Nvidia has great documentation on slicing up GPU based on time allotments to different workloads, but this is outside the scope of this series & indeed probably the needs of many individuals)

feel free to validate this with kubectl get deployments

vllm takes care of queuing, the OpenAI API cloning, pretty much everything you need to get started and worry less about the compute-heavy parts of the work. So in that spirit, we aren’t going to worry about it so much (but their documentation is great and the next tool we’re about to get prepared is coupled nicely with vllm specifically!)

Okay, now remove it with:

$ k3s kubectl delete -f vllm.yaml

Because in part ✌️ of this series, we’re going to create our “Control Panel” of our new LLM-chauvinist OS concept.

(it’s literally a control plane for the cluster with some tweaks)