Icinga stands tall in the realm of monitoring ecosystems, and it’s not just powerful — it’s a community-driven marvel, backed by the invaluable legacy of Nagios modules. It is the go-to solution for keeping tabs on the heartbeat of intricate infrastructures.

Imagine it as the vigilant guardian of critical systems, like the watchful eye over the detectors of the awe-inspiring Large Hadron Collider. However, navigating its bundled front-end rather complex for inexperienced users.

The icingaweb front-end might be powerful, but let’s face it — it’s not exactly user-friendly. In our quest for a simpler solution, we explored various approaches to deliver system statuses seamlessly to both web and mobile applications. But here’s the catch: access management was the thorn in our side. Who wants to install yet another app and remember additional login details just to check if everything is running smoothly? Our customers weren’t thrilled about the prospect either. They wanted a bridge between their devices and our treasure trove of monitoring data — something that didn’t involve app installations and password juggling.

Notifications with Telegram Groups.

At the same time Telegram notifications have always worked well because of the popularity of the messenger. Making icinga notification bot of your own is rather simple: go to BotFather, set name and description, grab a bot token then put bash scripts and add notifications from our repo to your icinga configuration:

conf.d/telegram-notifications.conf > /etc/icinga2/conf.d/
scripts/telegram-host-notification.sh > /etc/icinga2/scripts/
scripts/telegram-service-notification.sh > /etc/icinga2/scripts/

And here’s the gem of wisdom: when attaching notification groups, opt for the Telegram supergroup IDs instead of individual user IDs. Why, you ask? Well, it’s a brilliant strategy to maintain control while involving your customers and overseeing message delivery.

By associating notifications with the supergroup IDs, you can invite clients to relevant Telegram groups without relinquishing administrative privileges. It’s a power move that transforms access management from tedious icinga configurations to the visually intuitive process of Telegram group administration, so you can delegate these tasks outside of DevOps access scope.

Go to your Telegram web or app, create new group, go to group settings, click on ‘add user’, find your new bot and check it. Now write something to your group and check if bot is actually getting updates:

https://api.telegram.org/[TELEGRAM_BOT_TOKEN]/getUpdates

Find your message in the update list and copy ID of the supergroup. Now, edit telegram-notifications.conf and replace TELEGRAM_BOT_TOKEN with your bot token and TELEGRAM_GROUP_ID with the ID of your supergroup.

Interactive requests.

After completing these steps, we’ve successfully established a solid foundation for an interactive monitoring front-end based on Telegram. The next step involves updating our notifications bot to make it interactive.

To achieve this, we’ll use icingatelegram, a Nodejs application that serves as a clickable Telegram bot interface. This application connects with the Telegram API, acting as a webhook server, and interfaces with your icinga API to fetch monitoring data.

Icingatelegram

The bot interacts with your users directly through private chats. However, to initiate a conversation, users must first send a /sitrep request from the corresponding group to which they belong. The underlying assumption is that all group members have access to a specific set of Icinga services that you define in your configuration file. The group ID obtained from the initial request is utilized by the bot to determine the available dataset for the user and generate a session token. Now, your focus can shift to delivering targeted monitoring data upon user request to specific group IDs and Telegram will handle access control seamlessly.

Deployment as tgbot_swarm node.

tgbot-swarm

icingatelegram is designed to be deployed as Docker app with tgbot_swarm controller. To make it happen, follow these steps:

1. Prepare you tgbot_swarm node according to the manual.

2. Create API user on your icinga master host:

# /etc/icinga2/conf.d/api-users.conf

object ApiUser "icingatelegram" {
  password = "icinga-api-secret-password"
}

3. Make your own fork of icingatelegram repository.

4. Edit pipeline/deploy-icingatelegram.jenkinsfile according with you environment:

environment {
        // General application Configuration
        VERBOSE = "1"
        DEPLOY = "dev"
        RETURN_BUTTON = "1"
        LANGUAGE = "ru"

        // Application-specific deploy configuration
        APP_NAME = "icingatelegram-bot"
        APP_DESCRIPTION = "IcingaTelegram_monitoring_interactive_service"
        APP_HOME = "icingatelegram"
        // Telegram bot token from Jenkins secret store
        TG_TOKEN = credentials("icingatelegram-tgtoken-${DEPLOY}")
        // Monitoring API password from Jenkins secret store
        MONITORING_PASS = credentials("icingatelegram-monitoring-${DEPLOY}") 
        MONITORING_USER = "icingatelegram"
        MONITORING_API = "https://icingaweb.yourmonitoringmaster.org:5665/v1/objects/services/"

        // Swarm host-specific deploy configuration
        API_PORT = "8443"
        API_PATH = "controller"
        API_HOST = "0.0.0.0"
        API_KEY = credentials("swarm-apikey-${DEPLOY}")
        // Swarm node hostname Jenkins secret store
        SWARM_HOSTNAME = credentials("swarm-hostname-${DEPLOY}")
        // SSH Passwords for Swarm node from Jenkins secret store
        SWARM_SSH_CRED = credentials("swarm-sshcred-${DEPLOY}")

}

"service": {
    "-1234567891234": {
        "_alias": "Administrative Tech Group",
        "routers": {
            "name": {
                "en": "⚙️  Network Routers ",
                "ru": "⚙️  Сетевые Маршрутизаторы"
            },
            "type": "text",
            "endpoint": "SampleProject!services-network-routers"
        },
        "wifi": {
            "name": {
                "en": "🎚️  WiFi Hotspots",
                "ru": "🎚️  WiFi Точки доступа"
            },
            "type": "text",
            "endpoint": "SampleProject!services-wifi-hotspots"
        },
        "servers": {
            "name": {
                "en": "⚙️  Servers - Hardware",
                "ru": "⚙️  Серверы - Оборудование"
            },
            "type": "text",
            "endpoint": "SampleProject!project-servers-hardware"
        },
        "power": {
            "name": {
                "en": "🔌  Power Supply",
                "ru": "🔌  Электропитание"
            },
            "type": "text",
            "endpoint": "SampleProject!services-project-ups-all"
        },
        "printers": {
            "name": {
                "en": "🖨️  Printers",
                "ru": "🖨️  Принтеры"
            },
            "type": "text",
            "endpoint": "SampleProject!services-office-printers-toner"
        },
        "cpuload": {
            "name": {
                "en": "🧊  CPU Load",
                "ru": "🧊  Загрузка ЦП"
            },
            "type": "image",
            "endpoint": "http://graph.yourmonitoringmaster.org/S/a"
        },
        "memory": {
            "name": {
                "en": "🧊  RAM Usage",
                "ru": "🧊  Использование ОЗУ"
            },
            "type": "image",
            "endpoint": "http://graph.yourmonitoringmaster.org/S/b"
        }
    },
    "-3210987654327": {
        "_alias": "Private Tech Group",
        "server": {
            "name": {
                "en": "⚙️  Server summary",
                "ru": "⚙️  Обзор сервера"
            },
            "type": "text",
            "endpoint": "SampleServer!host-all"
        },
        "health": {
            "name": {
                "en": "⚙️  Health",
                "ru": "⚙️  Здоровье"
            },
            "type": "text",
            "endpoint": "SampleServer!host-nrpe"
        }
    }
}

5. Add Telegram bot token and icinga API login and password to Jenkins secrets storage:

icingatelegram-monitoring-dev: icinga-api-secret-password
icingatelegram-tgtoken-dev:  telegram-bot-token

6. Create new Jenkins pipeline and point it to deploy-icingatelegram.jenkinsfile in your repo.

7. Run the pipeline.

Note: icingatelegram stores session ID in the state memory, so your users will need to ask for access from the notification group after each time you run the build pipeline.

Installation as systemd service (CentOS).

If you wish to avoid using Docker and you are generally okay to dedicate port 443 of the host, you may set up icingatelegram as a systemd service on your host:

1. Install prerequisites:

yum install openssl git nodejs npm nano -y

2. Choose directory:

cd /opt

3. Clone repository:

git clone https://github.com/xyhtac/icingatelegram.git

4. Install Nodejs dependencies:

cd /opt/icingatelegram/icingatelegram
npm install

5. Create SSL key and certificate:

openssl req -newkey rsa:2048 -sha256 -nodes -keyout icingatelegram.key -x509 -days 3650 -out icingatelegram.pem -subj "/C=US/ST=New York/L=Brooklyn/O=Icinga /CN=icingatelegram-host-01.ydns.eu"

6. Create and edit bot configuration:

cp /opt/icingatelegram/icingatelegram/config/default.json /opt/icingatelegram/icingatelegram/config/local-dev.json
nano /opt/icingatelegram/icingatelegram/config/local-dev.json

{
    "defaults": {
        "verbose": "1",
        "return-button": "1",
        "defaultLang": "ru"
    },
    "telegram": {
        "host": "0.0.0.0",
        "port": "443",
        "token": "telegram_bot_token",
        "url": "https://icingatelegram-host-01.ydns.eu:443/",
        "key": "icingatelegram.key",
        "cert": "icingatelegram.pem"
    },
    "monitoring": {
        "api_url": "https://icingaweb.yourmonitoringmaster.org:5665/v1/objects/services/",
        "username": "icingatelegram",
        "password": "icinga-api-secret-password",
        "service": {
            "-1234567891234": {
                "_alias": "Administrative Tech Group",
                "routers": {
                    "name": {
                        "en": "⚙️  Network Routers ",
                        "ru": "⚙️  Сетевые Маршрутизаторы"
                    },
                    "type": "text",
                    "endpoint": "SampleProject!services-network-routers"
                },
                "wifi": {
                    "name": {
                        "en": "🎚️  WiFi Hotspots",
                        "ru": "🎚️  WiFi Точки доступа"
                    },
                    "type": "text",
                    "endpoint": "SampleProject!services-wifi-hotspots"
                },
                "servers": {
                    "name": {
                        "en": "⚙️  Servers - Hardware",
                        "ru": "⚙️  Серверы - Оборудование"
                    },
                    "type": "text",
                    "endpoint": "SampleProject!project-servers-hardware"
                },
                "power": {
                    "name": {
                        "en": "🔌  Power Supply",
                        "ru": "🔌  Электропитание"
                    },
                    "type": "text",
                    "endpoint": "SampleProject!services-project-ups-all"
                },
                "printers": {
                    "name": {
                        "en": "🖨️  Printers",
                        "ru": "🖨️  Принтеры"
                    },
                    "type": "text",
                    "endpoint": "SampleProject!services-office-printers-toner"
                },
                "cpuload": {
                    "name": {
                        "en": "🧊  CPU Load",
                        "ru": "🧊  Загрузка ЦП"
                    },
                    "type": "image",
                    "endpoint": "http://graph.yourmonitoringmaster.org/S/a"
                },
                "memory": {
                    "name": {
                        "en": "🧊  RAM Usage",
                        "ru": "🧊  Использование ОЗУ"
                    },
                    "type": "image",
                    "endpoint": "http://graph.yourmonitoringmaster.org/S/b"
                }
            },
            "-3210987654327": {
                "_alias": "Private Tech Group",
                "server": {
                    "name": {
                        "en": "⚙️  Server summary",
                        "ru": "⚙️  Обзор сервера"
                    },
                    "type": "text",
                    "endpoint": "SampleServer!host-all"
                },
                "health": {
                    "name": {
                        "en": "⚙️  Health",
                        "ru": "⚙️  Здоровье"
                    },
                    "type": "text",
                    "endpoint": "SampleServer!host-nrpe"
                }
            }
        }
    },
    "interface": {
        
    }
}

7. Try starting bot service manually:

npm start dev

Check console for errors and use Ctrl+C to stop nodejs instance.

8. Copy systemd service template:

cp /opt/icingatelegram/scripts/icingatelegram.service /lib/systemd/system/icingatelegram.service

9. Enable and run service:

systemctl daemon-reload
systemctl enable icingatelegram
systemctl start icingatelegram

10. Finally, create an API user on your icinga master host:

# /etc/icinga2/conf.d/api-users.conf

object ApiUser "icingatelegram" {
  password = "icinga-api-secret-password"
}

Icinga Aggregation Services.

Configuring your Telegram bot with individual buttons for each monitoring service might not be the most efficient approach, as your infrastructure is likely to comprise numerous services. The rule of thumb is to limit the amount of buttons to a maximum of ten to fit an average device screen.

The only effective workaround is to introduce aggregated services into your icinga configuration. Icingatelegram comes with a set of predefined aggregation services that help you to combine check results of hosts and services into one concatenated output, just like this:

PLUGIN OUTPUT

 OK: SERVICE DETAILS:
 * C_​BAK (snmp)​:
✅ snmp_​Disk C: Usage percent OK - 23 % 
✅ snmp_​Disk E - BACKUP: Usage percent OK - 76 % 
✅ snmp_​Physical Memory: Usage percent OK - 20 % 
✅ snmp_​cpu_​load: CPU Load percent OK - 0 % 

 * C_​FS (snmp)​:
✅ snmp_​Disk C: Usage percent OK - 75 % 
✅ snmp_​Disk E - STORAGE: Usage percent OK - 94 % 
✅ snmp_​Physical Memory: Usage percent OK - 18 % 
✅ snmp_​cpu_​load: CPU Load percent OK - 0 % 

 * C_​RDP (snmp)​:
✅ snmp_​Disk C: Usage percent OK - 88 % 
✅ snmp_​Physical Memory: Usage percent OK - 60 % 
✅ snmp_​cpu_​load: CPU Load percent OK - 3 % 

 * C_​DEV (snmp)​:
✅ snmp_​Disk C: Usage percent OK - 90 % 
✅ snmp_​Physical Memory: Usage percent OK - 49 % 
✅ snmp_​SAS_​E: Usage percent OK - 84 % 
✅ snmp_​cpu_​load: CPU Load percent OK - 1 % 

 * C_​PROD (snmp)​:
✅ snmp_​Disk C: Usage percent OK - 68 % 
✅ snmp_​Physical Memory: Usage percent OK - 41 % 
✅ snmp_​SAS_​R6_​F: Usage percent OK - 81 % 
✅ snmp_​SSD_​R1_​E: Usage percent OK - 82 % 
✅ snmp_​cpu_​load: CPU Load percent OK - 0 %

Set up aggregation services.

1. Add aggregation services to your icinga configuration. Service definitions are universal and should work well by default, no tweaks required:

conf.d/aggregation-services.conf > /etc/icinga2/conf.d/

Note: By default the aggregated service always return ‘Ok’ status ignoring all statuses of listed services. This is done to avoid notifications generation by the dummy host. If you prefer the listed service state to be propagated to the dummy host, uncomment state calculation blocks.

2. Add sample-project.conf to your Icinga configuration. Change project name, rename hostgroups according to your environment.

conf.d/sample-project.conf > /etc/icinga2/conf.d/

3. Add vars.type and vars.project to your host definitions to add them into aggregation hostgroups.

4. Restart icinga service for the changes to take effect.

systemctl restart icinga2

Now you can add aggregated services to the configuration of your bot, i.e. ‘SampleProject!services-office-printers-toner’ — you may copy this string from the URL field of your icingaweb on the service page.

Icingatelegram can also deliver images, so you can send graphite diagrams on request — just put graph URL to the ‘endpoint’ field and set ‘type’ : ‘image’.

• My main goal of building Commitcat was to convince HR-experts to refrain from assessing developer’s Github page as a performance metric. People who don’t know how to code should not evaluate a coder's discipline with a green calendar. 

• Now you can simply throw this link https://commitcat.webflow.io into anyone who brags about his commitment or questions yours. 

Bear at the party.

I love the Github commit widget, I find it a wonderful source of motivation and just a good example of an infographic. All these everyday-commit challenges and hackathons are just a holiday, hugging everyone, you are awesome.

The bad news is, folks, that the HR-people have come to our party. It all began with cute recruiter-girls, but now a bearded personnel officer with an onion fume and evaluation checklist in his hands is shaking his dandruff in the front door of your cozy GitHub.

We all know what they’re looking for out there. They are looking for a track record of your activity as a developer. They seem to already get that it is impossible to build up a solid GitHub history in an hour, unlike a good CV. But, of course, they don’t go beyond infographics, and according to an age-old HR tradition, they don’t spend more than 10 seconds on a candidate.

This news has been coming from the wilderness for, perhaps, more than a year, from different ends of geography. Being frustrated, I sketched out a code generator scheduler and made Commitcat. But I’m not here to sell you just another SaaS, It’s about our vision.

I don’t urge you to subscribe and use this thing, but I suggest you starting a flash-mob in the spirit of postmodernism: let’s post a link to this angry cat with a description wherever possible, so that the most junior HR person in every corporation would know for sure that the Github data is polluted. There is no point in looking at the applicant’s commit calendar, there is zero happiness in it.

In case of success, we will forget about this destructive practice of technical recruiting and regain inner peace. Why wouldn’t we give it a try?

Autocommit bot.

Speaking of CommitCat, it's not a mobile app or a web app — it's a bot on Telegram. If you don't have Telegram, just get on-boarded. Now, you have a bot that handles all your settings. It works with inputs and buttons as if it were a standalone application, and gives you a really good UX. User manual has only two simple steps:

1. Prepare new repository:

   

Go to your Github and create a new private repository. It will be generating a code-like rubbish, so it's better to keep it private. Don't forget to enable Include private contributions on my profile option on Settings / Public profile.

Next, go to Settings / Developer Settings / Personal access tokens and generate a fine-grained personal access token: set Expiration to a maximum of 1 year, choose Only select repositories in the Repository access section and select your newly created repo. In the Permissions section add Read and Write access for the Contents option.

Click Generate token and copy your new token.

2. Set up the bot:

   

No registration or account activation required, no applications to install and no password to remember.

Just go to 👤 Account / ⚙️ Settings  / 🦊 Connection section of the bot and set your Github username, repository name, your fine-grained token from Step 1 and your email.

Note that the email address will be used to sign your commits and it should match exactly with your Github user email.

If your configuration was accepted by my git client, you will see the 🟢 Activated status on the 👤 Account frame within next 3-5 minutes.

Under the hood.

Generator of Commitcat is rather simple. It imitates syntax of the language, makes random names for vars, classes and functions. It disregards logic, so the result rubbish can't be validated as a proper code. But as long as you put it in a private repo, who cares.

Meanwhile, I'm testing an LLM-driven code authoring module and it creates quite promising results. It can even produce some sane-looking stuff that could be released publicly if you’d need to.

The only downside is that it puts a significant stress on my infra. I decided to release it when the project hits at least 200 recurrent subscriptions, so I could reliably spread the expenses.

Alt gen module is based on llama3-70b with groq. Well running on my test rig, but its scaling price is considerably higher than conventional codegen.

Alternatives and ethics.

After the release I got comments on X with ethic concerns and descriptions for easier workarounds. Some addressed auto commit generation as spam, some noted that you don't need to do it every day — instead, you could run the script making commits with changed timestamps and have the whole chart filled overnight.

First and foremost, I am convinced that commits to your personal private repository cannot be considered a spam under any circumstance. The ideal solution, in my opinion, would be having an option on Github to silently randomize your chart. I think we will come there at some point.

Secondly, you can generate history for the past with a single transaction, true. But you will have to re-run this script every day if you want to keep up, and this is exactly what Commitcat is built for.

The whole point is not in actually running this kind of script and flooding your chart with unrelated data, rather in making everyone know that you can do it in a matter of single click.

Telegram offers JSON-based, accessible via a RESTful control webhook API updates to push data to the handler application running as a publicly available https server on one of valid ports (80, 443, 88 or 8443) with self-signed or CA-signed certificate.

A frugal build-and-forget methodology is oftentimes preferred to avoid the costs of using fancy off-site CI/CD platforms and no-code services. Popular and well-maintained open source bot libraries widely used by developers (python-telegram-bot, telegraf, etc) may serve several bot tokens per process, but it comes with a cost of combining multiple bot logic into one source repo.

So, here is the problem. Running multiple instances of dockerized bot applications on a single host considering a limited number of allowed ports requires nginx reverse proxy to take care of connection dispatch. Configuration of nginx has to be carefully tailored in regard with actual port settings of docker containers; nginx SSL-certificates also have to be consistent with all deployed bot applications. No external container orchestration (Docker Swarm, etc.) and no nginx control API (i.e. Unit) are allowed by the conditions of our task since we are about to fit one standalone host. Therefore, the amount of man work required to set-up and maintain single-host multi-bot dockerized infrastructure is significantly higher than the development process of the application itself.

One of the automated solutions comes with my tgbot-swarm CI toolkit.

To minimize efforts in bot deployment it solves two scopes of tasks:

1. Provide a built-in API that employs port, path and SSL-certificate assignment mechanism and a relevant proxy config generation. In a nutshell it is a node.js/express controller application that runs in the same docker container with nginx, serves RESTful JSON-based API and sends system control signals to the local nginx process.

2. Provide Jenkins groovy pipelines to automate build and remote deployment (using ssh in this example) of a controller/proxy container and an arbitrary bot container based on response from the controller.

And, as always, I would be more than happy to meet co-maintainers for this project.