<![CDATA[Lillychans Blog]]>https://blog.eps-dev.de/https://blog.eps-dev.de/favicon.pngLillychans Bloghttps://blog.eps-dev.de/Ghost 6.21Tue, 17 Mar 2026 15:52:23 GMT60<![CDATA[Setting up a VPN-only access for your Minecraft server using Pangolin]]>I always talk about the different ways to protect Minecraft servers from all kinds of danger. However, most of these arise from the fact that the servers are exposed to the public internet.

Most of the time having your server exposed to the public internet is the only practical solution.

]]>
https://blog.eps-dev.de/setting-up-a-vpn-only-access-for-your-minecraft-server-using-pangolin/697d2c1afde5ad0001c94412Sat, 31 Jan 2026 17:12:57 GMTI always talk about the different ways to protect Minecraft servers from all kinds of danger. However, most of these arise from the fact that the servers are exposed to the public internet.

Most of the time having your server exposed to the public internet is the only practical solution. However, there is another solution for friend groups and small communities which completely eliminates the public factor: VPNs

VPNs (Virtual Private Networks) simply allow all connected users to connect to each other over secure tunnels instead. This removes the necessity of having your server available to the public - and with that a large amount of the perceived danger.

While it would be possible to set up such a VPN completely yourself, it does involve a bit of trial and error. It also might not be as easy to manage for you. This is where Pangolin comes into play. Pangolin is a software which allows you to easily manage a VPN network and the users therein. It provides an easy to understand web GUI for configuration and user management.

In this blog post I want to deviate a bit from my usual style and instead provide a comprehensive Step-by-Step and Input-by-Input tutorial on how you could set something like this up for your friend group. (A tutorial on how to do the same with raw Wireguard will follow at some point)

Prerequisites

  • A domain you control
  • A cheap VPS
    • I can recommend Hetzner. Using my refferal link you get 20€ of credit for free to play around with.
    • It is assumed that the VPS is freshly set up and has some kind of Debian/Ubuntu running on it.

Step 1: Preparing the server

Make sure the software running on the server is up to date:

apt update && apt upgrade -y

Also ensure that curl is installed:

apt install curl -y

Pangolin recommends installing it via Docker, so we will first have to set that up. Luckily Docker provides a handy install script which we can use:

curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh

Now before we can set up Pangolin we have to prepare our DNS entries. Pangolin expects a wildcard entry to generate HTTPS certificates. For this tutorial I am going to use *.example.kittyscan.com but you can of course use whatever you want. Take note of your servers IP address and create the respective A and AAAA (if applicable) entries.

A screenshot of cloudflare. The two entries are visible as rows in a table.
The two required DNS entries

Step 2: Setting up Pangolin

The full documentation for this can be found here.

First we have to download the installer using this command:

curl -fsSL https://static.pangolin.net/get-installer.sh | bash

Then we can execute it by simply using:

./installer

You will see the following prompt:

Welcome to the Pangolin installer!
This installer will help you set up Pangolin on your server.

Please make sure you have the following prerequisites:
- Open TCP ports 80 and 443 and UDP ports 51820 and 21820 on your VPS and firewall.

Lets get started!

=== Basic Configuration ===
Do you want to install the Enterprise version of Pangolin? The EE is free for personal use or for businesses making less than 100k USD annually. (yes/no):

Select "no". The enterprise features are not necessary for this to work.

Enter your base domain (no subdomain e.g. example.com):

Respond with the domain you choose. In my case it is "example.kittyscan.com"

Enter the domain for the Pangolin dashboard (default: pangolin.example.kittyscan.com):

You can just keep the default here.

Enter email for Let's Encrypt certificates:

Enter your EMail there. This is required to generate the HTTPS certificates.

Do you want to use Gerbil to allow tunneled connections (yes/no) (default: yes):

Keep the default option (yes).

=== Email Configuration ===
Enable email functionality (SMTP) (yes/no) (default: no)

Also keep this on "no". You can set it up later if you want.

=== Advanced Configuration ===
Is your server IPv6 capable? (yes/no) (default: yes):

If your server has an IPv6 select "yes". If notn select "no". (Or get a better provider)

Do you want to download the MaxMind GeoLite2 database for geoblocking functionality? (yes/no) (default: yes):

Also just keep this at "yes".

Nearly done:

=== Starting installation ===
Would you like to install and start the containers? (yes/no) (default: yes):

Select "yes".

Would you like to run Pangolin as Docker or Podman containers? (default: docker):

Keep the default "docker".

Now it will pull and start the containers. Depending on your internet connection this could take a few minutes.

After it is done it will ask you if you would like to install CrowdSec. Just answer "no".

This will finalize the setup and tell you the next steps:

=== Setup Token ===
Waiting for Pangolin to generate setup token...
Setup token: 9z3rcy8hlujefrvw5w7y7068gbjfj09g

This token is required to register the first admin account in the web UI at:
https://pangolin.example.kittyscan.com/auth/initial-setup

Save this token securely. It will be invalid after the first admin is created.

Installation complete!

To complete the initial setup, please visit:
https://pangolin.example.kittyscan.com/auth/initial-setup

Visit the link it shows you. In my case it is "https://pangolin.example.kittyscan.com/auth/initial-setup". There you will see a page like the one below. Just enter your credentials and the token given by the installer. Once you have done that, just click the orange "Create Admin Account" button.

A screenshot of the setup form. All fields are filled in

You will be redirected to the login page. Enter your credentials and log in.

Now you will be asked to create an organization. Just give it a fitting name such as "Minecraft". Click "Create Organization" afterwards.

A screenshot of the "New Organization" form. All fields are filled in.

And with that the setup is done.

Step 3: Connecting our Minecraft server

Now that Pangolin is set up, we have to make our Minecraft server reachable through the VPN. For that click on "Sites" in the sidebar and then "Add Site". This site will represent our Minecraft server.

Keep "Newt Site" selected and enter a name for this connection.

On the bottom is a "Install Newt" section where you can find the commands for many use cases. In this case I will set up the Server via Docker Compose so I just copy the pregenerated config and click "Create Site".

Screenshot of the Pangolin interface. In the "Install Newt" sections Docker Compose is selected and on the bottom of the page is a docker compose file stub visible

A fully Docker contained setup using the itzg/docker-minecraft-server images now looks something like this:

services:
  mc:
    image: itzg/minecraft-server:latest
    environment:
      EULA: "TRUE"
  newt:
    image: fosrl/newt
    container_name: newt
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=https://pangolin.example.kittyscan.com
      - NEWT_ID=8mm9z4eruiihh5d
      - NEWT_SECRET=wk8p8lmrg1ay3mk6osbmiwqnfm1ciqew2z3p693wvi2gq2iw

Notice that we do not use any "ports" declaration. The "newt" container will connect to the VPN and provide access to the Minecraft server. No public availability needed.

Run docker compose up -d and after a few seconds you should see that the status has changed to "online".

Now navigate to "Resources > Private" and click "Add Resource". Here we configure who can access the server once they are within the VPN.

Enter a name for this resource. Make sure that the selected Site is the one you just created.

Under "Destination", keep "Host" selected and enter the name of the Minecraft container or some other way to reach it from your Site (localhost, ...). In our example it is "mc". Under "alias" you can enter a vanity DNS name that users can use to connect once they are logged in. Keep the port restrictions empty. Do not click "Create Resource" yet.

Switch to the "Access Policy" tab and under "Roles" select "Member". This will ensure that all users within the VPN can connect. You can also authorize selectively by selecting the accounts under "Users". Now you can click "Create Resource"

Step 4: Connecting our users

Now that our server is connected the only thing that is missing is ... us. To change that visit https://pangolin.net/downloads/ and download and install the client for your Operating system.

I am using Windows, but the steps should be roughly the same for all platforms.

Once you have installed the client it will be in your hotbar. Right click the icon and select "Login to Account"

Screenshot of the selection box.

This will open a window. Select "Self-hosted or dedicated instance". Enter the URL of your server, in our case "https://pangolin.example.kittyscan.com". Upon clicking "Login" a browser window will open prompting you to log in. Do that.

It will ask you if you want to authorize this device to log in. Select the "Authorize" Option. Once you see the "Device Connected!" screen you can close the browser window.

Now that you are logged in, right click the icon again and select "Connect".

Screenshot of the selection box. Now showing the "Connect" option.

Once the icon has turned orange you are successfully connected.

The only thing that remains now is to open Minecraft and connect via the name we defined at the end of step 3.

0:00
/0:09

It works!

To add your friends just create invite links in "Users > Users > Create User". Once they have created their account they can install the client the same way you have and start playing.

If you want to learn more or have any questions, feel free to join the KittyScan Discord server.

Join
]]>
<![CDATA[A Kitty year in review]]>The year 2025 is coming to a close and everyone is making one of these so I'll chime in.

2025 has been a truly moving year for me, with many ups and downs. Every success was accompanied by a fail. The more happy am I how the project

]]>
https://blog.eps-dev.de/a-kitty-year-in-review/694d8bbdaf604a0001189973Wed, 31 Dec 2025 15:57:40 GMTThe year 2025 is coming to a close and everyone is making one of these so I'll chime in.

2025 has been a truly moving year for me, with many ups and downs. Every success was accompanied by a fail. The more happy am I how the project I initially developed while having nothing to do with a 40°C fever has turned out. KittyScan is without doubt the biggest and most challenging project I have had to date. But also the most interesting in my opinion.

Before 2026 begins and new challenges are to be tackled I want to look back at the year and give an outlook for the future. (Spoiler: It includes Open Source)

Kitty Scan in numbers

Let's start with the raw numbers. In the nearly 9 months KittyScan has been running it has gathered a grand total of

  • 1.12M unique IPs running Minecraft servers
  • 3.27M unique players playing on these servers
  • 254K server icons

All of this amounts to a database which has grown to a total 150GB in size. The biggest table being the one holding a grand total of 219,673,945 datapoints for all the servers KittyScan has found - amassing to 103GB in size. You can read more about the troubles this has caused me in this blog post.

Our V1 honeypots have been contacted a total of 2,354,554 times. While our V2 honeypots that have been running for about a month now have been contacted 791993 times. These combined have come from a grand total of 9,302 IPs.

According to Google, the KittyScan Website had a total 12,752 impressions resulting in 2,807 clicks within the last 12 months. (I somehow managed to remove the robots.txt from the image so Google delisted us for a solid 3 months ... oops)

IPv6 is the future

... seems to be the "We are only 10 years away from fusion power" of the IT world.

But to be serious, IPv6 is still mostly unexplored when it comes to server scanning. This can be mostly attributed to the increased difficulty of the target generation part of scanning. While you can just scan the whole public IPv4 internet within an hour, that would be a multi million year feat for IPv6.

IPv6 scanning is an interesting research field. In the last month I already explored some options and launched KittyScanIPv6. Next year this will become a big part of my work, expanding the KittyVerse.

KittyScan and OpenSource

Nearly all KittyScan code is closed source. This will not change. I do not feel comfortable with sharing my tooling, even if it is easy to replicate.

However, starting next year I will start extracting parts of the KittyScan toolset into public libraries for everyone to use. These include functionality which I believe to be useful to others.

All libraries will be released under the MIT Licence, so that there are as few restrictions as possible on their usage. It is important to note that I refuse any contributions or feature requests for these. If you want to add functionality please just create a fork and maintain it yourself. I do not have the motivation to triage 3rd party input.

The first library is already public. KittyPrint is the utility I use to detect web services such as World Maps that are hosted on servers running Minecraft servers.

The next library will be a utility to download the spawn chunks of a given server - the same way KittyWorld does it. This is something I am currently developing to reduce my reliance on external libraries.

Thank you

I want to thank everyone that has supported the project over the last 9 months. Having such a great community helps with ignoring all the idiots such a project will inevitably attract.

I will stay focused on my goal of providing useful information to players and admins alike and raising awareness of the privacy and security caveats of running a Minecraft server.

Currently this project is entirely funded by me. If you'd like to support this project please consider subscribing to my Patreon you might even receive a Lillychan sticker pack.

But that was it for this year. I wish you a good transition into 2026 and a happy new year.

]]>
<![CDATA[Introducing KittyPaper]]>Since the beginning, the goal of KittyScan has been to make players and admins aware of security and privacy issues. KittyBlock was an attempt to create a simple solution to minimize the data that could be collected by scanners. Today I am introducing the next logical evolution: KittyPaper.

To maximize

]]>
https://blog.eps-dev.de/introducing-kittypaper/69370928745f3700017050f4Tue, 09 Dec 2025 13:50:52 GMTSince the beginning, the goal of KittyScan has been to make players and admins aware of security and privacy issues. KittyBlock was an attempt to create a simple solution to minimize the data that could be collected by scanners. Today I am introducing the next logical evolution: KittyPaper.

To maximize compatibility, KittyBlock was a Spigot plugin. Sadly this came with certain limitations. I could only modify the response and could not properly terminate the connection (I am excluding some hacky reflection solutions here). There also were some weird edge cases where the ping response was sent without ever being intercepted by the plugin. It also did not prevent the scanner from sending other requests for fingerprinting beforehand. In short: I did not have the authority I wanted.

So if a plugin does not allow me to do what I want to do, what is the next logical step? My own Paper fork.

Implementing the KittyBlock functionality, directly within the server software, allows me to filter and terminate connections before they ever were handed over to the server logic. This massively reduces the amount of possible work-arounds. Another benefit of this approach is that I can change the default values for the Minecraft/Spigot/Paper settings, allowing for more "secure" defaults . It will also allow me to implement some future™ features, which also require me to have a lower level of access.

At its current BETA stage, I have implemented the absolute minimal amount of features required to generate a practical value for players and admins.

Due to the privacy reasons outlined in this blog post, I have enabled hide-online-players by default. A setting has also been introduced which makes it so the player is always assumed to have forbidden being shown in the player listing if hide-online-players is not enabled.

The connection blocking has also been introduced. Any IPs which are found on the configured blacklists (KittyScanBlocklist by default) have their connections terminated before being handed over to the game logic. A dashboard is also provided which shows admins how many connections have been blocked. You can see a live demo for this dashboard here: https://kittypaper.com/dash/2c1a4412-cd34-4515-bee5-a0bb28c8d28c

Screenshot, Black backgroun   Select with the value "7 Days"   2 Boxes, Left one titled "Blocked Requests" with the description "The number of requests blocked by KittyPaper for this server" and a value of 2180, Right one titled "History" with a description of "... of blocked requests over time" showing a growing graph line.  Below there is a table showing a digest of the countries the requests where blocked from
Screenshot of the dashboard

Currently the KittyScanBlocklist has been generated solely using the data generated by my honeypots. KittyPaper allows this blacklist to be augmented using crowdsourced information. Any IP which pings the server, has never connected to the server or does not join the server within 30 minutes, is considered suspicious. (This behaviour can be disabled of course) There is currently no automated system for this argumentation in place, I will work on that once some data has accumulated.

It is important to mention that this does not prevent scanning and griefing outright, however it does greatly reduce the amount of exposure you have.

Being in active Beta, there may still linger some issues I have not found yet. Please report these on GitHub.

In the near future I want to add a comfortable way to manage whitelists.

I also invite other developers to adopt my changes as they see fit. All patches are licensed under MIT.

You can download KittyPaper here: https://kittypaper.com/download

Anyway that was it for this short announcement post. I will write some more about the technical background in the future.

]]>
<![CDATA[Admins, protect the privacy of your players!]]>By default, Minecraft servers publish a sample of their currently online players through their Status Response which is shown by the client when hovering over the player count field in the multiplayer menu.

Screenshot of the minecraft multiplayer menu. The player count is being hovered and shows "Lillychan_"
The player name, visible in the Minecraft client

Since this Status is given out to everyone who

]]>
https://blog.eps-dev.de/admins-protect-the-privacy-of-your-players/692c5d26730efb0001cf550cSun, 30 Nov 2025 16:35:26 GMTBy default, Minecraft servers publish a sample of their currently online players through their Status Response which is shown by the client when hovering over the player count field in the multiplayer menu.

Screenshot of the minecraft multiplayer menu. The player count is being hovered and shows "Lillychan_"
The player name, visible in the Minecraft client

Since this Status is given out to everyone who requests it, without prior authentication or authorization, server scanners like KittyScan can easily use this information to track players.

With the extreme amount of data server scanners accumulate that itself does not pose an issue. It does however allow for targeted stalking. Just knowing someones username allows for the creation of specific profiles, such as with whom someone is playing regularly, their usual playtimes, and many other details which can be extrapolated surrounding factors.

Since 1.18 players can decide to hide themselves from this sample.

By setting the Allow Server Listings toggle in Options > Online to off all servers this player joins are instructed to redact their username from the player sample.

By default, this is done by sending the username Anonymous Player combined with the UUID 00000000-0000-0000-0000-000000000000 instead of the real information.

Screenshot of the minecraft multiplayer menu. The player count is being hovered and shows "Anonymous Player"
The player name, redacted to "Anonymous Player" in the Minecraft client

This does not prevent anyone from joining a public server and just saving the player list, this would however be noticeable since it requires a bot to be on the server itself.

Since this setting is enabled by default and the client itself does not give any indication that this player sample exists, I fear that many players, who would prefer not being trackable this way, still are.

This is where server admins come in. Also since 1.18 the hide-online-players has been added to the vanilla server.properties file. Setting this to true will fully remove the player sample.

Since enabling this setting does no harm in any way other than removing that niche hover in the multiplayer menu, which I'd wager most people aren't even aware of, I highly recommend all admins to enable it. Especially if you run a community for commonly targeted or otherwise vulnerable groups. (LGBTQ+, your kids, ...)

As a general solution I'd like to petition this setting to be on by default, or to remove the player sample completely.

]]>
<![CDATA[The most average Minecraft name is "Saaeee"]]>Hey everyone, today I am back with a short blog post about Minecraft names.

A side effect of the crawling done by the KittyScan project is that, over time, we see lots of individual players through the "player sample" component of the Status Response given by the game

]]>
https://blog.eps-dev.de/the-most-average-minecraft-name-is-saaeee/6913606263362a0001764efcTue, 11 Nov 2025 17:34:32 GMTHey everyone, today I am back with a short blog post about Minecraft names.

A side effect of the crawling done by the KittyScan project is that, over time, we see lots of individual players through the "player sample" component of the Status Response given by the game servers. Even though this the curve of new players is slowing down a total of 2.68M players have been accumulated.

Graph. X axis showing the date (range of 04.03.2025 - today) and Y axis showing the total count of players. A slowing curve is visible
The count of newly found players is slowing down

In truth I have many more player entries in my database. A total of 34M as of writing this. However, we cannot assume all of these entries to be actual players. For example some servers use these samples to display some flavour text and servers in offline mode do of course also not return valid players.

This means that I have to validate these players before counting them towards the player statistics. This can be done with the official Mojang API. By giving it a list of UUIDs we get the corresponding username which we can compare to the values we got from the game servers. If these match, the entry can be seen as valid.

The Mojang API has strict rate limits, so it is of benefit to filter the implausible entries out beforehand. By validating whether the names follow the required rules for Usernames and checking if an entry contains an offline user we reduce the amount of requests necessary by a lot.

I will write about the privacy issues that arise from this player sample at a later point.

As a fun experiment I retrieved all currently validated player names and counted the characters in each of the 16 possible positions. Afterwards I just took the most used character for the positions and the result was that the average name is 6 characters long and is "Saaeee". Of course this is not a real name but it shows us that players like to start their names with a big "S" followed by the vowels "a" 0r "e" in some positions. With this methodology the most "uncommon" name is the 16 characters long "8QQQQQQQQQQQQQQQ", showing that names rarely begin with numbers and are unlikely to contain a capital "Q".

If we ignore the case most average name is still "saaeee" but the most uncommon one transforms into "868qqqqqqqqqqqqq".

At the moment these are over all names. The following table shows these averages just for the names with exactly the given length.

len count average average (low) not-average not-average (low)
3 3152 1e5 mUM 1ei h2m
4 35917 _oa_ soa_ UUUU 8686
5 90554 Saae_ saae_ 8QQQQ 8688j
6 204693 Saaie_ saaae_ 8QQQQF 8568qj
7 304730 Sanaie_ sanaie_ 8QQQQQQ 8888qqj
8 354992 Sanaaaee sanaaaee 8QQQQQQQ 8568qqqq
9 347220 Saneeaaee saneeaaee 8Q8QQQQQQ 85868qqqq
10 320424 Saneeaaaee saneeaaaee 9Q66QQQQQQ 966668qqqq
11 271994 Saaeeeaaaee saaeeeaaaee 7Q868QQQQQQ 7686888qqqq
12 236612 Saaeeeaaeaee saaeeeaaeaee 9QQ86QQQQQQQ 95986888qqqq
13 174282 Saaeeeaeeeiee saaeeeaeeeiee 9Q6687QQXQQQQ 986687888qqqq
14 136544 Saeeeeeeeeeiee saeeeeaeeeeiee 9Q9788QXQXQQqQ 9597887888qqqq
15 125986 Saeeeeeeeeeeiee saeeeeaeeeeeiee 8Q59857QXQZQQQQ 885985788qqqqqq
16 71753 Taeeeeeeeeeeeiee saeeeeaeeeeeeiee 8QQQQXXQQQQZQQQQ 858688888888qqqq

There is no real conclusion to this post. I just wanted to share this as a fun side-note. Have a nice day!

]]>
<![CDATA[The Azure outage from the perspective of a Server Scanner]]>All times in GMT+1

As you have most likely noticed, yesterday Microsoft Azure had a large outage. This is the second event of this scale in just a few weeks, as just a week ago AWS had the same thing happen to them.

I do not want to open

]]>
https://blog.eps-dev.de/the-azure-outage-from-the-perspective-of-a-server-scanner/69031b5d6ed1e60001b84668Thu, 30 Oct 2025 09:28:46 GMTAll times in GMT+1

As you have most likely noticed, yesterday Microsoft Azure had a large outage. This is the second event of this scale in just a few weeks, as just a week ago AWS had the same thing happen to them.

I do not want to open the discussion on whether it is good or bad that such outages cause halve the internet to collapse. Or the hyperreliance some companies have to these services. Instead I want to quickly detail how the outage looked and played out for me.

Since I am actively crawling Minecraft servers, my infrastructure and metrics are prominently linked to the Microsoft Azure where all official Minecraft services are hosted.

The first sign of the outage was from my Whitelist Scanners. These check if a given Server uses a Whitelist. Because an official Minecraft Account is needed to perform this check, they have to first authenticate with Microsoft and then perform the authentication handshake during the join process which requires communication with Mojang servers. At 16:41 I saw the first error message:

Post "https://sessionserver.mojang.com/session/minecraft/join": dial tcp 13.107.213.67:443: connect: connection refused

That itself is not out of the ordinary, as I get similar results from time to time, mostly when they update if I had to guess. But then it continued, from 16:44 onward every single join attempt resulted in the same error:

Post "https://sessionserver.mojang.com/session/minecraft/join": dial tcp 13.107.213.67:443: i/o timeout

My first instinct was that my IPs got blacklisted (again). While this should not happen anymore as I seem to have found a system which prevents the automatic protections to kick in, I could not exclude the possibility of them having made their filters more aggressive. So I shut the whitelist crawlers down for the day.

It wasn't until an hour later when I played around with Minestom that I noticed, that I could not authenticate. That is when I first noticed the big news.

So what do the metrics say? Let's first look at the "Online Players" graph. This graph details the amount of players that have been found in the player sample reported by servers within the last hour, that have been confirmed against the Mojang API. While this metric is far from perfect and should not be seen as an actual player count, it can still give an indication for player activity over time.

Graph titled "Oniline Players" X axis shows the time and the Y axis shows the player count, a range of the last ~3 days is selected. A regular pattern is disruped by a sharp drop.

Looking at this graph we can see that the player count reaches its daily peak around 22:00. The projection was just in line yesterday until 16:00 where it began to sharply drop, reaching its lowest point at 20:20. Shortly afterwards the player count recovered back to its usual count at 23:00.

The player count which is self-reported by the servers shows the same general trend, however it does not quite recover until today.

Graph titled "Oniline Player Distibution" X axis shows the time and the Y axis shows the amount of server with a given player count, a range of the last ~3 days is selected. A regular pattern is disruped by a sharp drop.

This was all I wanted to share. Thank you for reading and have a nice day!

]]>
<![CDATA[Tales from optimization hell]]>With the ever growing dataset of my KittyScan project came unforeseen performance issues. How I ultimately fixed them, and how I made the process unnecessarily hard for myself is what I want to discuss in this short blog post.

Let's first look at the use case these issues

]]>
https://blog.eps-dev.de/tales-from-optimization-hell/68fe95d8f41e750001f9cd8fTue, 28 Oct 2025 18:11:47 GMTWith the ever growing dataset of my KittyScan project came unforeseen performance issues. How I ultimately fixed them, and how I made the process unnecessarily hard for myself is what I want to discuss in this short blog post.

Let's first look at the use case these issues appeared in. The scanning infrastructure of KittyScan is divided into two components, a "head" and the "scanners". The head is the service responsible for distributing jobs (eg. "scan this IP", "check if this server uses online mode", ...) to the scanners which execute these jobs and respond with the result. These two components are connected to each other with a RabbitMQ bus, for load balancing and buffer reasons.

With scanners just blindly working through their jobs, the head has to process the responses and persist them in the database.

While the real database schema is a lot more complex, the only relevant detail you need to know is that it has a table called "servers" which contains all basic information about a given server at a given time, such as the MOTD or the player count. This table is already deduplicated by only saving deltas between scans to minimize the entries generated. If I would not do this, the table would grow by around 190k rows per hour. During the roughly 7 months of runtime, this would have amassed 957 Million entries. Using deltas reduced this to only about 100 Million entries, a reduction of 90%. Nice.

Because of this delta check, the head has to compare the current result against the one last saved in the database for the given IP. And because the head receives about 190k answers per hour, it also has to do this check 190k times per hour.

So how do I get the last saved state from the database? Just filter by the IP and sort descending by the (incrementing) ID. This results in a query which looks something like this:

SELECT * FROM servers WHERE ip_id='0.0.0.0' ORDER BY id DESC;

And while this is the most straight forward "head through the wall" approach to this issue, it did work surprisingly well. For the first few months this ran without any problems. Or so I thought. Over time I noticed a small but linear decrease in ingest speed (the rate at which the head consumes the answers given by the scanners). While it started of as "as fast as you can throw stuff at it" it slowly but surely declined down to ~70 per second. This wasn't an issue at the time and I chose the message bus explicitly to allow for buffers to build up, but in time this would not keep up. So I had to look for a solution.

Before the delta check I have to query some other tables to collect all necessary data (Players, Favicons, ...). So my first thought was that doing all these queries onto also growing tables would be the root cause for the reduction in ingest speed. Luckily all of these tables have entries which are added to, but never edited or removed. A favicon, once saved, will never change or disappear for example. This made these queries perfect for a simple in memory cache, which only contacted the database if the item was not yet queried before. This was a little bit tedious but simple. And it was effective. With the caching implemented, the ingest speed rose to ~300 per second again.

Problem solved. Not.

About a month later the ingest speed had again fallen down to ~70 per second. So I took another look. I saw the query above often took ~250ms to finish. That was something that confused me in the beginning, but after some trial and error I noticed that I forgot to give the "ip_id" field an index. That was a stupid mistake, which caused the Database to search in the raw data instead of an optimized lookup table, but it was easy to fix. Also, as a side note, I want to applaud Postgres here for still being that fast, even with a mistake on that level.

So I quickly added that index and, look at that, the query only took ~40ms and ingest speed was up again, not as much as before but good enough. It also did not fall in any capacity I would have noticed.

That was until a few days ago, when I moved all my services to a new server. The server had much more performance and much better I/O, so color me surprised when I saw what the ingest speeds were down to ~5 per second and the query took ~4000ms to finish.

Panik.

My first attempt to fix this was to REINDEX the database, maybe the indexes built by pg_restore were faulty. And yes, it got better after this, but only a bit. The ingest speed was up to ~60 per second again which is barely enough to prevent a backlog from building. I knew that I had to fix this or else the project may not be able to continue.

I searched for every bit I could optimize and found nothing.

Then, when looking at a completely different part of the software, it hit me. I had, a long time ago, already built a 1:N table between IPs and the last server found them. I did this for a different microservice and forgot about it afterwards. The most ironic thing about that is that this 1:N table is being updated by the same ingest function which also encompasses the delta checks.

Using this table to reference the latest entry instead of the query speeds things up by a ton. The time to retrieve the data whet down from ~40ms to below 0.1ms.

But curiously after implementing this fix, I still saw the same messages about the query taking ~400ms, which confused me even more, I thought that I had maybe forgotten to push my changes. But nope. It turns out that I somehow did not manage to run this inefficient query once per ingest but twice.

The second query was originally not part of this ingest function but after I refactored the head a bit to simplify some processes it ended up there and I never considered just reusing the result I got from the first one.

And with that fixed we are back to an ingest speed of "as fast as you can throw stuff at it" again. Hooray.

So what is the lesson I should learn from this? Firstly is to not stop optimizing every time it reaches "good enough" again, if I have the time I should invest it in looking for further improvements. Secondly is to monitor my metrics more closely to identify performance issues faster and not just randomly. And thirdly is to know that I should avoid relying on database "magic". While it does certainly help, it may be lost during the next migration to something else. To be entirely honest, I did not even know that Postgres did so much optimization in the background. But maybe this also just was a happy collection of coincidents.

Welp, that was it for today. Thank you so much for reading and have a nice day!

CTA Image

Join the KittyScan comunity today!

Discord

]]>
<![CDATA[50% of Minecraft servers do not use a whitelist - and that's a problem]]>KittyScan is a crawler of mine which searches the internet for Minecraft servers, with the goal of getting some interesting insights into the Minecraft ecosystem and how it evolves over time.

KittyScan gathers this data by scanning the public IPv4 range for Minecraft servers running on the default port (25565)

]]>
https://blog.eps-dev.de/50-of-minecraft-servers-do-not-use-a-whitelist-and-thats-a-problem/68668a44cb5d480001351468Thu, 03 Jul 2025 18:46:37 GMTKittyScan is a crawler of mine which searches the internet for Minecraft servers, with the goal of getting some interesting insights into the Minecraft ecosystem and how it evolves over time.

KittyScan gathers this data by scanning the public IPv4 range for Minecraft servers running on the default port (25565). Because of this behaviour most larger or complex server setups are not found. But since I want to focus on the smaller friend servers instead of the larger networks, this is not an issue. But it's something to keep in mind when reading the stats.

Disclaimer: Due to several restrictions no dataset will be perfect.

I will write some more about KittyScan and its findings later on. Today I want to focus on an issue that is bigger than I ever could imagine. Recently I started probing the subset of servers which run with "online mode" enabled and a vanilla-like software (Mojang, Spigot, Paper, ...) by automatically joining them with an online account to check whether I get kicked for whitelist reasons or am able to join without issues. By now I have reached a substantial sample size and a stark trend becomes obvious: More than 50% of Minecraft servers do not use a whitelist.

I could look at it in a positive light, that nearly 50% of servers use one. But that is just not enough. If I only look at the servers which have not been customized, so the ones most likely to be used by individuals and their friends, that number approaches 70%.

But why is that an issue? It's a game server after all, not something important. With a disabled whitelist, any player with an Minecraft account could join. And if there are no precautions against it, they can grief. There are near daily posts on Reddit from players that lost their world due to various griefing groups. Often there are no backups. But the way bigger issue is, that if there is any security flaw found in the future, especially one on the scale of the 2021 Log4J Remote Code Execution exploit (CVE-2021-44228), there are tens of thousands of Minecraft servers open to exploit.

The main issue I have with this is that Minecraft servers are delivered "simple by default". We have to accept that the average person hosting a Minecraft server themselves or via a provider, is more likely to be an amateur than someone hosting a PostgresDB. They are less likely to know some security best practices, they are less likely to know how to create backups. And I'd argue that that's fine. Everyone had to start somewhere. But I firmly believe that the software should be preconfigured with this in mind. It should be "secure by default".

But I think there is another component to the issue. A big chunk of the players starting up servers do so to play. They might not know anything about server administration. They might not know that there is a whitelist. A lot of players might feel nervous towards using commands. And nothing tells them how it works. They have to proactively search and learn about it.

Mojang and the developers of the various Bukkit forks cannot influence everything about the environment a server is installed in. But there is one crucial part: enable the whitelist by default. Besides having online mode enabled, an active whitelist is the second best protection the default server software can offer. This would be a breaking change. But the technically adept admins often already use prepared config files so they could also adapt for this change if it is clearly communicated. But for a player without prior knowledge this would be perfect.

Same thing for server providers: preconfigure them with the whitelist enabled by default, offer a small UI where your customers can add their friends with the click of a button. You might also add "Secure By Default - AI Supported Whitelist Management" to your selling points for all I care.

Instruct new admins to add players to the whitelist, give them example commands, take them by the hand. The option to disable the whitelist should not be explicitly mentioned. This would ensure that every new server has had a fair chance to be configured with a whitelist. And if the admin decides to disable it that's totally fine as long as it is an informed decision.

Mojang even links to this very useful setup guide on the server download page, sadly most players will just straight up ignore the link and download the server file, or more likely get their server executable from a source which is not minecraft.net.

I know that there are plugins which allow the whitelist to be managed by Discord verification and what not, but all of them are more complex to use than the default whitelist system for small friend servers

This is how I imagine the first startup of a fresh server to look like:

Starting net.minecraft.server.Main
[...]
[Server thread/INFO]: Time elapsed: 2295 ms
[Server thread/INFO]: Done (4.414s)! For help, type "help"
[Server thread/INFO]: 
[Server thread/INFO]: The whitelist is enabled but does not yet contain any players.
[Server thread/INFO]: Add your first player by running "/whitelist add <playername>" 
[Server thread/INFO]: Learn more about how to secure your server here: https://minecraft.wiki/w/Tutorial:Setting_up_a_Java_Edition_server#Security_recommendations

That where all thoughts about this that I have at the moment. It's just a very important issue for me, because it breaks my heart to read how some players just lose years of progress because of this. And I'd plead for the "heh should have used a whitelist and made backups 5head" finger pointing towards these players to stop. This kind of toxic behaviour helps no one. I also want to clarify that I do not blame Mojang or any Minecraft developer. These are just my two cents.

Anyway I'll continue procrastinating on building more stuff for the KittyScan website. Bye.

]]>