A Docker/LXC/Kubernetes, database-free, lightweight container performance monitoring solution
972
A Docker, LXC, database-free, lightweight container performance monitoring solution, perfect for ephemeral containers (e.g. containers used for DevOps automatic testing). Can also be used with InfluxDB and Grafana to monitor long-lived containers in real-time.
The project is composed by 2 tools:
The agent is actually a cgroup-aware statistics collector; cgroups (i.e. Linux Control Groups) are the basic technology used to create containers (you can read more on them here); this project is thus aimed at monitoring your LXC/Docker container performances but can equally monitor physical servers.
This project supports only Linux x86_64 architectures.
Table of contents of this README:
This project collects performance data about:
cpuacct (CPU accounting) cgroup;memory cgroup;blkio cgroup;Moreover the project allows you to easily post-process collected data and produce a self-contained HTML page which allows to visualize all the performance data easily using Google Charts.
You may be thinking "yet another monitoring project" for containers. Indeed there are already quite a few open source solutions, e.g.:
All these are very complete solutions that allow you to monitor swarms of containers, in real time. The downside is that all these projects require you to setup an infrastructure (usually a time-series database) that collects in real-time all the statistics and then have some powerful web platform (e.g. Graphana) to render those time-series. All this is fantastic for persistent containers.
This project instead is focused on providing a database-free, lightweight container performance monitoring solution, perfect for ephemeral containers (e.g. containers used for DevOps automatic testing). The idea is much simpler:
If you use an LXC/Docker container based on a Centos/RHEL/Fedora distribution you can log into the container (or change its Dockerfile) and just install the RPM right away from the COPR repository:
yum install -y yum-plugin-copr
yum copr enable -y f18m/cmonitor
yum install -y cmonitor
If you use an LXC/Docker container based on a Ubuntu distribution you can similarly install from my Ubuntu PPA using the following commands:
add-apt-repository ppa:francesco-montorsi/ppa
apt-get install cmonitor
If you want to simply use a out-of-the-box Docker container to monitor your baremetal performances you can run:
docker pull f18m/cmonitor
which downloads the Docker image for this project from Docker Hub. See below for examples on how to run the Docker image.
The RPM installs a single utility, cmonitor_collector inside your container; launch it like that:
cmonitor_collector -s 3 -m /home
to produce in the /home folder a JSON with CPU/memory/disk/network stats for the container
sampling all supported performance statistics every 3 seconds.
Whenever you want you can either:
cmonitor_chart utility to convert that JSON into a self-contained HTML file (mostly useful for ephemeral containers);
see below for practical examples.To plot the JSON containing the collected statistics, simply launch the cmonitor_chart utility installed together
with the RPM/Debian package, with the JSON collected from cmonitor_collector:
cmonitor_chart /path/to/json-stats.json <optional HTML output filename>
Note that to save space/bandwidth you can also gzip the JSON file and pass it gzipped directly to cmonitor_chart.
In this case you can simply install cmonitor as RPM or APT package following instructions in How to install and then launch the cmonitor collector as any other Linux daemon. Example results:
baremetal1:
example of graph generated with the performance stats collected from a physical (baremetal) server running Ubuntu 18.04;
note that despite the absence of any kind of container, the cmonitor_collector utility (like just any other software in modern Linux distributions) was running inside the default "user.slice" cgroup and collected both the stats of that cgroup and all baremetal stats (which in this case mostly coincide since the "user.slice" cgroup contains almost all running processes of the server);
baremetal2:
This is a longer example of collected statistics (results in a larger file, may take some time to download) generated with 9 hours of performance stats collected from a physical server running Centos7 and with 56 CPUs (!!);
the cmonitor_collector utility was running inside the default "user.slice" cgroup so both "CGroup" and "Baremetal"
graphs are present;
In this case you can install cmonitor Docker using the official DockerHub image, see Docker; the Docker container will collect all performance stats of the baremetal. Just run it
docker run -d \
--name=cmonitor-baremetal-collector
-v /root:/perf \
f18m/cmonitor
Example results:
If you can easily modify the Dockerfile of your container, you can embed cmonitor so that it runs inside your container and monitor your dockerized-application. Example of the Dockerfile modified for this purpose:
...
COPY cmonitor_collector /usr/bin/ # first you need to grabthe cmonitor binary for your Docker base image
CMD /usr/bin/cmonitor_collector \
--sampling-interval=3 \
--output-filename=mycontainer.json \
--output-directory /perf ; \
myapplication
...
Example results:
cmonitor_collector instance that monitors the performance of the Docker container itself;
in this case both cgroup stats and baremetal performance graphs are present.In this case you can simply install cmonitor as RPM or APT package following instructions in How to install and then launch the cmonitor collector as any other Linux daemon, specifying the name of the container it should monitor, e.g.
docker run --name userapp myuserapp-docker-image
# here we exploit the following fact: the cgroup of each Docker container
# is always named 'docker/container-ID'
cmonitor_collector \
--sampling-interval=3 \
--output-filename docker-userapp.json \
--cgroup-name=docker/$(docker ps --no-trunc -aqf name=userapp)
Example results:
The cmonitor_collector can be connected to an InfluxDB deployment to store collected data (this can happen
in parallel to the JSON default storage). This can be done by simply providing the IP and port of the InfluxDB instance when launching
the collector:
cmonitor_collector --remote-ip=1.2.3.4 --remote-port=8086
The InfluxDB instance can then be used as data source for graphing tools like Grafana which allow you to create nice interactive dashboards like the following one:

You can also play with the live dashboard example
To setup easily and quickly the chain "cmonitor_collector-InfluxDB-Grafana" you can checkout the repo of this project and run:
make -C examples regen_grafana_screenshots
which uses Docker files to deploy a temporary setup and fill the InfluxDB with 10minutes of data collected from the baremetal.
This project started as a fork of Nigel's performance Monitor for Linux, adding cgroup-awareness;
but it has quickly evolved to a point where it shares very little code pieces with the original njmon tool.
Some key differences now include:
cmonitor_collector and plotted by cmonitor_chartcmonitor_collector;cmonitor_chart differently organized;cmonitor_collector is able to connect to InfluxDB directly and does not need intermediate Python scripts to transform
from JSON streamed data to InfluxDB-compatible stream.This fork supports only Linux x86_64 architectures; support for AIX/PowerPC (present in original nmon) has been dropped.
Just like the original project, this project is licensed under GNU GPL 2.0.
Content type
Image
Digest
sha256:58b645d62…
Size
7 MB
Last updated
over 1 year ago
Requires Docker Desktop 4.37.1 or later.