And all of this took us to the point where we can list all the services that are not part of the infrastructure, that are truely services… I use Homepage as my main dashboard. I configured it with traefik ingress route annotations watcher. If I remove a service, it will be automatically removed from the dashboard, same thing if I add one, it will be auto.

All the Infra and Infrastructure categories are already covered in the series, monitoring too

Vaultwarden

I think that presenting Vaultwarden is not needed, I put all my passwords in here, use it on the computer, the iPhone, the iPad and it is an amazing software. I was waiting for so long that Bitwarden finally commit account switching to the client because I was using the vault of my work with everything in one. When the commit was finally merged recently I deployed my own vault and now use it for my personnal accounts. I have no problem using the compagny one as only you can unlock your vault but you know the urge of SELFHosting everything.

Miniflux

I use miniflux everyday to track my RSS feeds, I also track github releases. My choice went for this one because the webui works on mobile, it has postgres support, and OpenID connect.

Vikunja

I do all my TODO and tasks management with Vikunja that also support postgres and OpenID connect. I found the user experience nice even there is some little bugs but the dev is really reactive when you report them !

It-tools

This one is on the list of my daily drivers for sure. A nice collection of tools that comes handy when your are dev or sysadmin !

Pdf processing

Stirling PDF let you Add / Remove / Rotate / Reorganize pages in a PDF, and many more things !

Online copy / paste

Private Bin gives me the possibility to send password or sensitive data to someone with the “Burn after reading” option, If the person tell me that he can not guest the pass, I know it is compromised.

Document Management System

Paperless NGX watch a share on my NAS and archive everything that I scan and send to that share via the office scanner. I am really bad keeping papers… Now I am bad at scanning papers but I made a step forward…

Ambient sound to cool your head off

Moodist is an very simple service to deploy and act as a sort of ambient sound jukebox, you can set your own audio mood and let it run in background.

But you prefer to blast your eardrums with AC/DC like me?

Deemix is a cool tool to download music from a Deezer account and you can play it after in Navidrome ( not on the dashboard )

Url shortener

Shlink is my URL shortener, I used it with private bin before noticing a major security flaw… Now I keep it but do not use it at the moment. It provides a nice GUI to manage your links and some insights.

Google alternative

This one is the start of something I want to try for a long time : self host as many privacy oriented alternatives as I can. I currently looking at Invidious for Youtube, Wikiless for wikipedia, … you get the idea.

Whoogle is one of them, OK it fetches results from Google but remove a ton of crap before serving them to you…

Wetransfer alternative

I found Pingvin Share as a self hosted alternative I use it quite often. But I also came across Sharry and I found it more robust, I will make the swap I think.

Documentation

Before writing this is started to document everything in WikiJS and I need to continue. Very nice UI / User experience, everything is stored in postgres so the pods are stateless and you can scale them juste by increasing the replicas count. Also support OpenID.

Youtube downloading

If I need to download a song / video / playlist from youtube I go to my own instance of Metube.

File sharing

I still use filestation for now but Nextcloud is the Goal to replace it soon.

Linux Distrib ISOs sharing

Still use downloadstation for that, I had a previous VM with Deluge and I think I will try to go with it again, worked quite well in a previous lab

Sketching

All my schemas are done with my own instance of Excalidraw

So how does it looks ?

And that’s a wrap for my 2024 Homelab tour !, as I already said reach me on reddit for questions…

Long Live r/homelab & r/selfhosted!!!

Authentication

After deploying the first services I wanted to add authentication everywhere because some webUI like Longhorn or Thanos do not implement auth. At the beginning I did basicauth with my traefik ingress but I quickly searched for a true Single Sign On solution. My criteria were :

• Forward auth compatible to use it with traefik
• All in one solution
• OpenID provider
• FIDO keys support

So I tried authentik then found it complicated for nothing, and this BUG…

Then I found Authelia and use it now for everything. It is running in front of the services that needs auth because they do not provide it themselves. Services that are compatible with OpenID use it as a provider, that way I can login one time and acces everything. I use Yubikeys as my second factor.

All my authelia users are stored in a LDAP database, I use LLDAP to manage them, it is a very lightweight ldap implementation that is perfect for that use case.

I use the following middleware in traefik for the services that can not do OpenID :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: forwardauth-authelia
  namespace: authelia
  labels:
    app.kubernetes.io/instance: authelia
    app.kubernetes.io/name: authelia
spec:
  forwardAuth:
    address: http://authelia.authelia.svc.cluster.local/api/verify?rd=https://whatever.yourdomain.com/
    authResponseHeaders:
      - Remote-User
      - Remote-Name
      - Remote-Email
      - Remote-Groups

Configuration management

I used Ansible several times for work and found it amazing. We also leverage AWX to manage / launch our playbooks stored in GIT but for the Homelab I wanted to try to little brother of AWX : Semaphore. Features parity is not here, but for the homelab it will cover my needs.

The CI / CD Stack

GitOps

As said in Part 1, I have a Gitea instance ( soon to be replace by Forgejo ) that store every single YAML files deployed in my Kubernetes Cluster. I use the GIT as a single source of truth for the cluster. I will try to extend that beyond the Kubernetes cluster, with my VM on the proxmox nodes and with Ansible.

CI

I use Woodpecker as my CI tools to build this website for example and some container Images. Nothing fancy here, juste a standard woodpecker install that runs in Kubernetes and launch pods for each steps in the pipeline. I use Kaniko to build docker images in kubernetes and push them in my private registry.

CD

ArgoCD is the tool that sync everything between my GIT repo and my cluster. I deployed it in HA mode with the YAML files of the project. With autosync and autoheal enabled it will fix the cluster if the state drifts from the desired one that is stored in Git. There is an additional tool that you can use : ArgoCD Image Updater, that will update the image of your app if it finds a new one in the registry.

But asides from that Image updating part, ArgoCD is a wonderfull tool that also provide a nice view on all your apps :

I use it to deploy plain YAML files, Kustomize files and I recently found that you can deploy a Helm chart from a repo with values from another repo so I will migrate all my Helm chart to ArgoCD ! I should be able to get rid of the image updater addon with using Renovate bot… I also love the fact that your ArgoCD apps objects can be deployed as plain YAML too because they are CRDs, Argo store everything in Kube and do not need a Database or persistent volume :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/allow-tags: regexp:[1-9]+-[1-9]+
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/image-list: yourprivateregistry.domain.com/hugo/hugo-blog
    argocd-image-updater.argoproj.io/update-strategy: name
    argocd-image-updater.argoproj.io/write-back-method: git
    notifications.argoproj.io/subscribe.on-sync-succeeded.teams: geco-teams
  name: hugo-blog-prod
  ...
spec:
  destination:
    namespace: hugo-blog-app
    server: https://kubernetes.default.svc
  project: default
  source:
    path: hugo-blog/overlays/prod
    repoURL: https://yourgitrepo.domain.com/h.campion/hugo-k8s-manifests
    targetRevision: HEAD
  syncPolicy:
    automated:
      selfHeal: true

You can also see there the annotations used to config to image updater addon and the teams notifications on succesfull sync.

Cloud Native Postgres

For sure one of my favorite tool of the stack… And this one was suggested to me by someone on reddit, thanks for this one mate, I do not know your username anymore sorry…

CNPG is the solution I use to deploy all my databases. I went with postgres because of that operator, it just works, and without single failure since the start for me… And trust me I broke it everyway possible at the start and never lost a single bit of data… CNPG will do all of this for you:

• Bootstrap a replicated PGSQL cluster with 3 members in my case
• Init the DB and the user
• Init from a previous backup if you need to
• Create a scheduled backup of your DB to a S3 storage
• Create the key / certificates combo used to connect to your database
• Store them in a regular kube secret to let you use it in your app
• Deploy an exporter to let you watch you cluster with prometheus.
• Selfheal your cluster if you destroy a member…
• More cool shit that I’m not aware of…

It works well with ArgoCD as you can definie your DB cluster as YAML files like so :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

---
# Example of PostgreSQL cluster
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: vaultwarden
spec:
  backup:
    barmanObjectStore:
      destinationPath: "s3://cnpg"
      endpointURL: "https//your_s3_compatible_endpoint:9000"
      s3Credentials:             # The credentials of your s3 bucket
        accessKeyId:
          name: minio-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: minio-creds
          key: ACCESS_SECRET_KEY
  instances: 3                   # Number of members you want in that cluster
  primaryUpdateStrategy: unsupervised
  storage:
    size: 10G
    storageClass: local-storage  # As explained before,
    # I use local volumes as CNPG will manage
    # HA on its own and I want max IO performance for the database...
  bootstrap:
    initdb:
      database: vaultwarden
      owner: vaultwarden
      secret:
        name: vaultwarden-db-secret
  monitoring:
    enablePodMonitor: true    # Tell cnpg to enable prometheus monitoring

Monitoring stack

High Available Metrics Gathering

I use Prometheus and Thanos deployed in Kubernetes with the kube-prometheus-stack. It ease the setup of :

• Prometheus operator that will deploy and configure Prometheus and AlertManagers
• Prometheus instances, 3 in my case, with local volume like CNPG
• AlertManagers Instances
• Sidecars for the prometheus instances that Thanos will need
• Kube State Metrics which is a Kubernetes exporter
• Node exporter deployed as a DaemonSet on all kube nodes
• All the needed CustomRessourceDefinitions
• Grafana to view dashboards

I found this HelmChart very good, you will need a good amount of time to read the values file and customize it for your needs but the default is killer to get started fast. I added a lot of exporters to the defaults ones :

• Every linux server outside my cluster with the Node exporter
• Powerdns cluster metrics exported by recursors and authoritative servers + piHole exporter
• Blackbox exporter to monitor HTTP metrics of my services
• PushGateway that is used to receive asynchronous metrics like K8UP job completions
• SNMP Exporter to poll SNMP on my NAS and Unifi devices, switches…
• Speedtest Exporter for internet bandwidth checks

I also try to gather metrics from all the services that supports it by leveraging ServiceMonitors or PodMonitors:

• Prometheus / Thanos components metrics
• Loki components
• Grafana
• Authelia
• Traefik
• CNPG clusters
• Healthcheckio status
• Longhorn
• Argo and woodpecker
• …and everything that can expose a /metrics endpoint ^^

On top of that I use thanos to be able to query my 3 prometheus instances, and ship the data to S3 compatible minio on my NAS. I keep one week of data on the prometheus instances, in case of NAS failure I have 7 days of metrics to look for if needed. Then I keep one year on the NAS with the compactor downsampling the data to 1 hour after 30days. In my every day usage, I do my queries with the Thanos webUI, and look alerts on it too. My Grafana is also connected to the Thanos query frontend.

I used the bitnami chart for Thanos to help with the config of :

• Thanos querier that will fetch metrics from Sidecars and Storage Gateway
• Thanos query frontend that enhance the query path with caching and retry
• Thanos compactor that will compact / downsample TSDB block in your s3 storage
• Thanos storage gateway that make your S3 storage available as a datasource for the querier !

Visualize the data

Of course Grafana is used because it is the best Open Source visualization product on earth !

In kubernetes your can configure it to be autoprovisionned from configMaps, that way the datasources and the Dashboards are kubernetes objects that can be stored in my Git and deployed with ArgoCD. I gathered some dashboards from internet, some comes directly from helm-charts or operator, and some from me. I’m currently in the process of migrating my own dashboards to configMaps to make my Grafana “stateless”.

Here are some example :

Checking alerts

Healthchecksio

To monitor my cron tasks and backups that are not in Kubernetes I deployed Healthchecksio. In two words : a service that will wait for the process you want to say Hello ! on a defined schedule and send you alerts when it does not…

Website analytics

This website use Umami to do analytics. I considered Matomo too but the lack of postgres support made me give up the idea. I did not want to deal with two database operators just for Matomo. Umami is also simpler / lighter. Initially I was using a HUGO theme that was not compatible with Umami so I hid the script snippet in the footer but since I switched to a Umami compatible theme I should remove that and use the regular config.toml…

Logging stack

I’m still testing this on my setup because I have issue with storage right now ( space ) that I have to migrate but my previous tests where good. The setup I am evalutating is Loki installed in scalable mode with the official helm chart. The grafana agent is installed in Operator mode and use Promtail ( the loki agent ) under the hood to gather logs from my Pods. I managed to gather the acces log of my Traefik ingress controler without problem.

I used the same technique as CNPG and Prometheus for storage, local persistent volumes to have maximum performance. Logs are all shipped by Loki to my NAS via S3 protocol.

More on this later…

PVC Backups

I use K8up to backup all my kubernetes persistent volumes.

Oversafe backup strategy…

Objects that I backup :

• Host themselves with the Proxmox backup client CLI in a cronjob => to the pbs server
• Virtuals machines config and drive with => to the pbs server
• Kubernetes PVC with K8UP => to my NAS
• CNPG Databases => to my NAS
• Longhorn PVC with longhorn backup system => to my NAS
• Configurations => to my GIT that is a saved vm to the pbs server

The nas is snapshoted several times a day as I use btrfs and I sync it to an offsite NAS every evening.

The proxmox backup server is also synced to offsite PBS everyday.

The backups are watched by prometheus or healthcheckio and send teams notifications in case of failure. I also see the failure on karma or grafana.

Let’s now talk about other services

The fundation

The cluster consists of 6 Virtual Machines running on the Proxmox Cluster:

• 3 Control Plane node with 2 Cores and 6G of RAM
• 3 Worker node with 6 Cores and 20G of RAM

At the time I’m writing this, they are running Debian 11 as OS and Kubernetes 1.28. I will update them to Debian 12 as I already started migrating some machines to 12 and the Hypervisor are already on 12. But for Kubernetes I like to stay one version behind, as they support the 3 last version, there is no problem doing this.

The cluster is provisionned and updated via the Kubeadm tool, I wanted to learn the Kubernetes “Vanilla” tool, then I found it quite simple so I ditched to idea to switch to K3S… even this product is quite cool too !

It is in HA Mode with Etcd as backend, CoreDNS for the cluster DNS and I use kube-vip in ARP mode to share a Virtual IP between my control plane nodes. I can always access the API is one node still up !

I use Calico as my CNI ( Networking ) plugin, deployed with the Calico Operator, but I want to try Cillium

Each proxmox host one worker and one control plane node.

Storage

Longhorn

I wanted to have shared storage between my nodes, but not by using my NAS as a single point of failure. So I use my NAS to access data on it via NFS Mounts for Musics and Media that are not used by critical workloads, but for things that I do not want off when my NAS is in maintenance, I use Rancher Longhorn. As my workers are connected through their own bridge with 10G, longhorn is taking advantage of the faster network.

Longhorn data are on a dedicated disk on the VM, this disk is on the nvme storage of the proxmox. I can case longhorn fill up the disk, the workload using longhorn will crash but the kube node will not…

Local volumes

I also use local volumes for some workloads when I prefer to manage the availability directly with the product… The most common use case I have for that is Postgres Databases. I do not need to have replicated storage as the replication is managed by the database itself… I also use this for Prometheus metrics for example because my Prometheus is in HA Mode too, more on this later !

This is the storage class I use for that :

1
2
3
4
5
6
7

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

And if I want a volume :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-local-worker3-privatebin
spec:
  capacity:
    storage: 10G
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/postgres-volume/privatebin # Path on the host you want to use
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-worker3

LoadBalancer

To deploy Loadbalancer type service I use Metallb in ARP mode. My services are tied to a virtual IP and Metallb will manage that VIP to make it follow the service when the pods are moving. The downside of this mode is that all traffic is routed to one node, so it can not be considered as true loadbalancing, but for failover it works like a charm !

Ingress Controller

The Ingress controller of that cluster is Traefik because when I started to learn docker and swarm I found it perfect, so I tried it on kube and found the same level of happyness with it ^^. I use the Custom Ressources Definitions from Traefik but the “regular” ingress api object is also enabled so that other software can leverage traefik like helm, operators or cert-manager.

I also use it as my reverse proxy for non-kubernetes workload. My NAS, Hypervisor WebUI, PBS WebUI are also served by traefik. To do this I use external service defined “by hand”.

Here is the proxmox webUI example which use 3 servers for backend :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: v1
kind: Service
metadata:
  name: external-pve
  namespace: traefik
spec:
  clusterIP: None   # We set clusterIP to none as the service is OUTSIDE
  ports:
    - protocol: TCP
      port: 8006
---
kind: Endpoints
apiVersion: v1
metadata:
  name: external-pve
  namespace: traefik
subsets:            # Then we set the endpoints "by hand"
  - addresses:
      - ip: xxx.1
      - ip: xxx.2
      - ip: xxx.3
    ports:
      - port: 8006

Then the “normal” Traefik IngressRoute to use it :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: external-pve
  namespace: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`xxx.domain.com`)
      kind: Rule
      services:
        - name: external-pve
          port: 8006
          scheme: https         # We use https because the proxmox webui listen on TCP 8006 with TLS
          sticky:
            cookie:
              name: traefik-sticky # And we set stickyness to make the NoVNC console work !
  tls:
    secretName: xxx.domain.com

Addons

There is some Addons that I think are really usefull in a Kube cluster.

Kube metric server

The Kube Metrics Server is here to give you basic metrics on your cluster, it is needed for the top commands to works like so :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
manager1   176m         8%     3566Mi          61%
manager2   170m         8%     3828Mi          65%
manager3   249m         12%    3548Mi          60%
worker1    978m         16%    14335Mi         71%
worker2    1247m        20%    15032Mi         75%
worker3    3362m        56%    11942Mi         59%

kubectl top pods
NAME                       CPU(cores)   MEMORY(bytes)
traefik-78d8f67fc8-scpv8   6m           115Mi
traefik-78d8f67fc8-sxb4z   13m          220Mi

But this is also what you need for Horizontal and Verticial Pod Autoscaler to work ! This is less true now because I’m pretty sure that I saw you can connecter thoses autoscaler to custom metrics providers.

Sealed Secrets

SealedSecrets is mandatory if you want to go full GitOps like me, it will allow you to push secrets to a git repo, even a public one, because your secrets become encrypted and only your kubernetes cluster can decrypt them ! It works like this :

• You create a “regular” kubernetes secret in mysecret-unsealed.yaml
• You encrypt that secret with kubeseal -f mysecret-unsealed.yaml -w mysecret.yaml
• It will output a file to mysecret.yaml
• kubectl apply -f mysecret.yaml
• The kubeseal controller will see the sealedsecrets object in your cluster and will unencrypt it to a regular secret that you can use like any other k8s secret !
• You can ditch the unsealed file if you want, in any case that file should NOT be commit to a git repo ! The sealed one is made for that !

I use it for every secrets, all my secrets are commited to a git repo, I also use a .gitignore with a pattern in case I forgot unsealed files somewhere…

Kured

Kured is as Good as simple… It watches your nodes ( deployed as a DaemonSet ) for a specific file and reboot them when that file is present. So as I have debian with automatic upgrades configured, it will reboot the node when new kernels are installed for exemple. You can configure a lot of things like :

• Planning, when nodes are allowed or not to reboot
• Where to send notification on reboot, I use teams
• Only drain pods that match a label
• A Prometheus instance to abort reboot if alerts are found
• Max nodes to reboot at the sametime …

Defrag Cron Jobs

I use to have regular warnings of my Etcd Database being fragmented so I created a cronjob to take care of this and never had anymore alerts :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-defrag-cronjob
  namespace: kube-system
spec:
  schedule: "00 16 * * *"
  successfulJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: etcd
              image: k8s.gcr.io/etcd:3.5.3-0
              securityContext:
                seccompProfile:
                  type: RuntimeDefault
              args:
                - /bin/sh
                - -c
                - etcdctl='etcdctl';
                  export ETCDCTL_API=3;
                  etcdctl --endpoints="https://IP_OF_A_MANAGER_NODE:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --cert="/etc/kubernetes/pki/etcd/peer.crt" --key="/etc/kubernetes/pki/etcd/peer.key" defrag --cluster;
              volumeMounts:
                - mountPath: /etc/kubernetes/pki/etcd
                  name: etcd-certs
          nodeSelector:
            node-role.kubernetes.io/control-plane: ''
          volumes:
            - hostPath:
                path: /etc/kubernetes/pki/etcd
                type: DirectoryOrCreate
              name: etcd-certs
          restartPolicy: OnFailure
          tolerations:     # Tell kube that this pod can run on managers !
            - key: "node-role.kubernetes.io/master"
              operator: "Equal"
              value: ""
              effect: "NoSchedule"
            - key: "node-role.kubernetes.io/control-plane"
              operator: "Equal"
              value: ""
              effect: "NoSchedule"

Cert Manager

Cert-manager allows you to request / issue certificates automatically. Several projects use it so I guess everyone needs it in a cluster at some point. Unless you manage certs yourself. I also use it to request HTTPS Certs for Traefik… I use the HTTP challenge with Let's Encrypt.

Compliance checker

I found this little tool called Popeye than can scan a cluster and tell you where you made some not so best practices things. There is an option to output in HTML format… So I use it directly in cluster, as a CronJob, and expose the result with a Caddy Web server pod. This way I can check the compliance of my cluster with a web browser.

Diun Image checker

Diun will watch your cluster and send alerts when new version of Images you are using got a update. As I’m running full GitOps I will switch this to Renovate but this little cool tool is worth mentionning !

Next part will be the supporting services running on that cluster

Let’s start with the physical part

Hypervisors

I run 3 HP SFF Desktop machines as Proxmox node, I bought them used. Original lab was done with little MFF Dell optiplex, 3050 with core i5 and 32Gb ram each, performance was good with very low power consuption but they were running hot… The nvme especially, so I decided to abort it and went with bigger one to have better temperatures and 10G networking !

So there is a 3 nodes cluster, storage is local with lvm thin on a RAID1 of 2 nvme, no HA, I manage availabily at service level.

I use them to run QEMU Virtual Machines and LXC Containers but I’m migrating anything that left on LXC to QEMU. I use Debian OS everywhere I can !

I use two separate bridges for networking, one with the two gigabit cards in LACP, with every VLAN tagged on it. All VMs / LXCs use it, backup go through this one, the WebUI too. The Two 10G cards are on a separate bridge, no LACP. only Kubernetes workers use this one and I made a loop with the main switch. The switch use the spanning tree to avoid making a mess and I can loose one cable or one host and still be able to reach every node. I use dell Direct Attach Copper cables for this loop, some Reddit research showed me that DAC are less expensive and run cooler !

Backup server

The oldest of the stack, nerver failed me…It runs Proxmox Backup Server and all vzdumps from Proxmox hosts are stored here !

Due to the fact that the HP BIOS won’t let you boot on the CDROM SATA port, where the SSD is connected, I installed GRUB on a small USB stick that seats inside the server, I boot from this USB, everyting else is on the SSD ( OS ) , ZFS RAID10 Storage ( backups ) on the Spinning drives.

Nas

This is my main storage for personal documents, media, backups… The only thing that run on this is DSM obviously and a minio docker container because I need S3 compatible storage. And of course NFS / SMB for the shares. My kubernetes cluster use NFS to mount volumes.

Networking

ISP WAN Piece of Shit

Getting rid of this is on the roadmap, I will try a ubiquiti GPON… In the meantime it is providing net to my main router and all ports are forwarded to it… All filtering is disabled, no DHCP, DNS or WIFI !

Main Router

More information of this amazing board Here. This is my main firewall, only purpose of this box is filtering, routing, NAT and VPN. By default everything is rejected both ways IN and OUT, I try to be as precise as I can to just open what is needed. There is two VPN on it, one wireguard for me when I’m not at home and need access to something that is not exposed. And a LAN to LAN openvpn with the society I work for ! This way I can work from home, send my backups off site, and we can send the society backup in my Homelab, win/win for everybody ! Futur project is to have two off these sharing virtual IP to do HA.

Switching

SW1 Core switch : Mikrotik CRS326-24G-2S+

Other switches

SW2 Unifi Flex Mini

This little gut is Amazing, for 26€ you get a 5 ports Gigabit swith that have VLANs, what else ?

SW3 Unifi SW8

Last switch that I use close to the TV, it connects to WIFI AP and some media equipments to the rest of the house. This one is fully functional but more expensive…

Unifi AP Pro

Only one Access Point with 3 SSID, one for the LAN, one for the GUESTS clients and the last for IOT devices. I live in a flat so only 1 is more than enough !

VLANS

The network is segmented with VLANs:

• LAN : There is everyting in here, this is the main one.
• IOT : The shit vlan, smart tv, smart light bulbs, things that I do not trust !
• GUEST : This one is for the GUEST Wifi SSID, it gives only internet.
• STORAGE : PVE and PBS are connected in this one for the backups.
• ADMIN : Switches, Access point, PBS & PVE Webui / SSH
• CLUSTER: This one is for corosync, the Proxmox cluster communication protocol

Base services

DHCP Cluster

My overkill DNS Architecture…

This one could be an article for itself. I use two PiHoles for DNS filtering / blacklisting, two PowerDNS Recursors and two PowerDNS Authoritative Servers connected to a master/slave MariaDB database !

My DHCP cluster pushes the two piHoles to my clients, servers are using the recursors directly. I have 2 000 000+ domains on blacklist, with the time I’ve got to the point that a good amount of crap is blocked and I’m not bothered anymore with false positives… The PowerDNS solution is very robust, the cluster keeps my DNS service running even if one of the two vm is down for maintenance. I can route my queries where I want, this is really important for my work, and I can gather some metrics for my monitoring ! I also installed PowerDNS Admin webui, nice way to manage my records, I quite like it to be honnest.

Unifi controller

Used to manage Unifi devices, push updates, checks connected clients…

Git

This is a Critical One, I use it to store :

• This website source
• GitOps for all the kubernetes Yaml / Kustomize manifests
• Ansible Playbooks
• Sometimes a project I need to fork to make a PR

It is a Gitea instance for now and it will be switched to Forgejo !

Registry

I use Harbor to store container images. CVE Scanning is enabled and we can see that the image for that blog have warning… I’ll take care of that !

Docker cluster for tests

I have a 3 nodes Docker Swarm cluster running with Portainer installed.

This is used as a testing environment for me, at work we still use Swarm a lot and if I want to quickly test something it is easier on this. There is one manager ( No HA needed ) and two workers.

Minecraft server

I still have an Old Minecraft server that I used to play on, still here because I do not want to destroy the map on this one. I do not know anymore how this thing is working but I remember that it had a management interface in CLI to launch command directly on the server…

CA

This one is a little special, I needed a PKI to manage my certs / CA, and at work we use our PfSense for that… I found this dead simple and great so I installed a PfSense VM, stripped every service I can and use it as a PKI with a WebUI… For example, ldap server certs, openvpn certs, … It use less than 256M of RAM…

Honney0gain worker

I have a honneygain worker, it basically use your internet connection to make money, they give you 0.0000000001% of what they earn and they possibly doing shaddy things of my back… So this is the perfect exemple for the IOT VLAN… I will throw this shit away but they promised me 20 bucks so I have to reach the goal before…

And that is the End of the Base / Physical part, let’s talk about the kubernetes cluster now !