Pyro Engineering

Modrinth Servers February Release

Amy — Mon, 10 Feb 2025 00:00:00 GMT

New Features

You can now search and filter through your server's console in the Overview tab, jump to specific results to see the log in context, select them, and copy them.
You can now drag and select any number of lines in the console, copy them. and view them formatted.
Hide your server's modrinth.gg custom URL using the new Hide subdomain label toggle in Options > Preferences.
The Content page has been updated to make managing your server's mods and plugins easier than ever. Now, only versions that are available for your server's Minecraft version and platform are shown by default, and you can now show beta and alpha versions in the selector.

Improvements

The Overview page loads faster.
The Options > Properties page loads faster.
The server hardware graphs in the Overview page have been rewritten to improve power efficiency and fix rendering bugs.
The modpack selector in Options > Platform now shows more information about a modpack, like its tags, downloads, and followers.
Reinstalling your server no longer requires the browser to refresh the page in order to work properly. We now also lock more options while a server installs to prevent your server from bricking itself.
The server console has been rewritten to implement proper batching. All performance issues with the console previously have now been fixed.
An error state has been added in the server list if servers are unable to be fetched.
Sorting in the Files tab is now accessible by clicking the column headers.
Backing up a server and erasing all its data simultaneously in the Platform page now works as expected.
Opening a platform modal, then opening another, no longer causes versions of that platform to fail to load.

Thanks everyone for your support, your feedback, and your patience. We work 24/7 to keep your servers running smoothly and to make sure we have capacity for all of you. As always, if you ever need help with anything simply send us a message through the chat in the bottom right corner of your server's panel and we'll get it fixed ASAP!

Modrinth Servers Security Incident

Pyro — Fri, 24 Jan 2025 00:00:00 GMT

We have posted our transparency report detailing the incident and our response. The post below remains for historical purposes.

What happened?

On January 20, 2025, a malicious threat actor gained unauthorized access to Pyro's infrastructure platform via a compromised GitHub Personal Access Token. During the breach, the actor had access to the database containing Modrinth Servers customer data. Modrinth user data (hosted by Modrinth) and Modrinth Servers data (hosted by Pyro) are separate from each other, and no part of Modrinth, like user data, billing information, and content were compromised.

The threat actor had access to server names server IDs, server IPs and ports, server subdomains, general server metadata (mod loader, installed modpacks, Minecraft version), backups metadata, and SFTP credentials, which have been reset for all Modrinth Servers. Three customer servers were directly accessed by updating the owner to a Modrinth account controlled by the threat actor. We have proactively contacted the customers affected and have already secured their servers.

As of today, January 25, we have fully resolved this security incident. We are no longer experiencing, nor expect to experience, any operational disruption to Modrinth Servers. We have no evidence that there is any malware or continued unauthorized activity within the platform. Outside of database access and certain dangerous APIs which were disabled immediately in response, the threat actor's access was limited at every step of the incident. As a result, the vast majority of customers data were not accessed by the threat actor. Pyro will be releasing a transparency report, including a full timeline of events, root cause, and a detailed log of our security response, within the week.

All Modrinth Servers customers will have their service extended by two weeks at our expense as a result of this incident. Please contact support if you have any further concerns and our team will get back to you right away.

Modrinth Servers Security Incident - Transparency Report

Pyro — Fri, 31 Jan 2025 00:00:00 GMT

On January 20th, 2025, Modrinth was alerted that three users' servers had been compromised, seemingly by the same person. Initially, it appeared their accounts had been hijacked, however, closer inspection revealed a deeper compromise of our infrastructure. In the following days, more holes were punched through Pyro's systems, leading to the attacker mass renaming backups and gaining access to our production servers and private GitHub repositories. We've hardened our security systems as a response, and as of January 25th, 2025, have no evidence the attacker has access to any part of our infrastructure.

The attacker had access to server IDs, IPs, ports, server metadata (server names, backup names, modloader information), server data, and our private code repositories. The attacker did NOT have access to billing information, Modrinth accounts, or Modrinth's infrastructure.

With the goal of providing transparency after this incident, we'll explain how our systems operate, detail the course of events, and outline the steps we have taken to mitigate the damage and reinforce our security.

How Our Infrastructure Operates

Pyro runs all servers on NixOS, a system designed for reproducible environments. Every server uses nearly identical configurations to simplify updates and maintain consistency. These configurations are stored in a private repository we call supercluster.

Historically, to manage deployments, we relied on a special user, “robot.” This user had privileged (root-level) access to our management server, allowing us to push new configurations rapidly across all machines. While useful for quick rollouts, this design meant anyone possessing the robot user’s credentials could gain full access to our infrastructure.

Early Clues of a Breach

The first sign of trouble appeared on January 20th, when a user reported that someone had taken over their server. Despite helping them secure their account by resetting passwords and enabling two-factor authentication, we noticed repeated break-ins. Even more concerning, the control panel showed the server’s owner had changed to an unknown Modrinth account, locking the rightful user out.

We initially believed this to be a one-off incident, perhaps caused by a compromised user password, but the facts soon suggested otherwise. By January 21st, investigation revealed no API route or typical user-level exploit that could account for such persistent ownership changes. Our suspicion of a larger infrastructure breach prompted a broad internal review of logs, endpoints, and code.

Initial Mitigations

Our database used to be self-managed on the same server our backend API is running from. In an attempt to shield our critical data, concerned the attacker had gained database access, we decided to migrate it to Neon DB. The assumption was that moving the database offsite, along with updating and revoking key credentials, would isolate or at least contain any ongoing attack. We increased backup frequency, hardened database access through IP whitelist and ensured our NeonDB account was secured.

Nevertheless, more reports of hijacked servers came in, confirming that something far more serious was happening. By January 23rd, it had become clear we were dealing with a sophisticated intrusion rather than a simple user account compromise.

Further Evidence of a Large Scale Breach

On January 24th, all doubts were erased when we discovered that more than 3,000 server backups had been renamed to racist slurs within the span of a single minute. No minor exploit or user-end weakness could account for such an immediate and large-scale alteration. It was unmistakable: the attackers had access to our database.

At this point, we had rotated our internal secrets several times already, such as the backend API's master key. We locked down our APIs, specifically deletion endpoints, limiting the damage the attacker could deal and stopped pushing updated secrets to our repositories.

The Real Culprit: A Compromised SSH Key (and a Hidden Token)

After digging in our management server, it became evident the "robot" SSH key had been leaked.

We found a work-in-progress software meant to automate NixOS deployments—an unfinished system not yet ready for production. In order for this software to clone and update certain repositories, it had a plain-text, unscoped GitHub Personal Access Token (PAT) stored locally, granting full read access to our private repositories, including supercluster, where many credentials and keys lived unencrypted. Every time we rotated database passwords or API keys, the attacker re-pulled supercluster using the stolen PAT to discover the latest information.

We came to the conclusion the attacker used the leaked SSH key to gain access to our management server, found the insecure PAT, and used it to keep control over our infrastructure.

Containment and Cleanup

Realizing that the unscoped PAT was the real “open gate” into our infrastructure, we immediately revoked it, along with any other lingering personal access tokens. We severed the “robot” user’s SSH privileges, regenerated new keys, and removed the incomplete deployment tool from the management server altogether.

Once that pathway was closed, the attacker could no longer reacquire our secrets, effectively halting their infiltration. We then launched a thorough internal audit to ensure no other unexpected tokens or processes were left behind. Each server’s logs were scrutinized, suspicious files removed, new monitoring and alert systems put in place, and we kept a close eye on SSH logs. Since these measures were enacted, there have been no further signs of unauthorized access.

On January 25th, the attacker reached out to us, requesting 1 ETH within the hour (eventually lowering to 0.25 ETH, and extending the delay by multiple hours) in exchange for not leaking Pyro's private codebases, our CEO's private codebases, and doing further damage to our infrastructure. At that time, there was no evidence the attacker had access to our infrastructure despite claiming otherwise. We have not, and will not, pay the attacker.

Lessons Learned and Steps Forward

This incident has highlighted many points of failure in our security systems, such as:

The "robot" SSH key granting root access to every node in our network.
Improperly stored secrets, acquirable if any infrastructure team member had their GitHub account compromised.
Poor internal diligence with handling critical secrets.
Absence of a proper response plan for security incidents.

It also sparked a greater conversation about security at Pyro and Modrinth: we are working closely together to ensure such a breach never happens again and detailing a response plan for future security events, including improving our transparency and response effectiveness.

As of now, the following improvements are being implemented:

Personal Security Overhaul: Every infrastructure team and management team member went through their personal security and ensured they were protected against targeted attacks, including resetting our computers, rotating critical passwords, ensuring 2FA is enabled everywhere and securely storing our SSH keys. Future measures may include storing keys exclusively in hardware tokens, such as YubiKeys.
Individualized Credentials: We've replaced the "robot" key with unique access for each trusted user, and are working on solutions to additionally restrict and log SSH access.
Vaulting Secrets: Keeping secrets in plaintext is unacceptable. We're now storing them encrypted only and ensuring only trusted people have access to them, and working on further solidifying secret access through idP and SSO software. Furthermore, secrets are scoped exclusively to the nodes that require them, preventing a leak of management secrets on customer nodes.
Hardened Monitoring: Through our analysis during the security incident, we noticed many areas where we lacked logging and monitoring. We should've noticed the malicious SSH logins and database modifications sooner. We're evaluating our options to integrate improved monitoring and runtime security tooling with our existing alerting systems, and implementing full auditing and centralized logging across all nodes.
Cultural Changes at Pyro: This incident highlights our lack of focus on security and safety. We're determined to engage in thorough security review within our company in the future, including:
- Detailing a security emergency response plan alongside Modrinth, with steps to take in the case of an intrusion such as revoking secrets and immediately notifying our customers.
- Rotating secrets and ensuring previous Pyro employees lose all infrastructure access to avoid disastrous leaks.
- Undergoing 3rd-party security auditing the moment it is financially viable for us.

Ever since this incident, our primary focus has been on security. We'll post a follow-up, including further steps we took to protect our customers, early next month.

Supporting Our Community

We recognize that some users experienced a total loss of control over their servers, and others saw vile, hateful changes to their data. For this, we extend our deepest apologies. We have taken steps to restore every compromised server and remove any malicious alterations such as renamed backups, and fully refunded affected customers who wished so.

Modrinth and Pyro are extending everyone's subscription by 2 weeks, free of charge. This means your next server bill will be delayed by 2 weeks. No action is required to receive this.

If you notice any irregularity with your server or simply have concerns about the security of your account, please reach out to our support team. We remain committed to transparency, and we promise to keep you informed if any further revelations come to light.

While this incident posed a serious challenge to our entire platform, it has reinforced our determination to maintain a resilient and secure environment for the Modrinth community. We are grateful for your patience and will continue to work diligently to protect our users, their servers, and their data.

— The Pyro Engineering Team

Our Journey and the Future of Pyrodactyl

Elizabeth — Mon, 12 Aug 2024 00:00:00 GMT

import Figure from "../../components/posts/figure.astro";

Our Journey

At Pyro, we began with a simple yet powerful belief: open-source software can reshape industries and empower communities. This belief became the cornerstone of our mission to revolutionize hosting. From the start, we aimed to create the most advanced fork of Pterodactyl, pushing the boundaries of what’s possible in hosting.

Our journey was challenging. We spent over six months meticulously rebuilding the core components of the panel. Every line of code was scrutinized, every feature optimized, and every pixel designed with precision. As Pyro grew, so did our vision. We realized we were no longer content with refining what already existed; we wanted to break new ground.

This drive led us to explore new technologies and methodologies, challenging the legacy systems that had long constrained the industry. The Pterodactyl fork was our launching pad, but it was not our final destination.

The Decision to Relicense

In alignment with our core values of transparency and community empowerment, we have chosen to relicense Pyrodactyl under the GNU Affero General Public License version 3 (AGPLv3). This decision reflects our deep commitment to open-source principles and ensures that Pyrodactyl remains free and accessible.

We chose the AGPLv3 license because it:

Mandates that modifications, improvements, or derivative works are shared with the community
Ensures the hosting industry continues to benefit from collective advancements
Prevents the monopolization of innovations emerging from Pyrodactyl

Our choice of AGPLv3 came after carefully considering how best to protect Pyrodactyl’s integrity while encouraging widespread collaboration. We wanted to create a framework where developers, companies, and enthusiasts could contribute to the project, knowing their work would benefit the entire community.

Empowering the Community

To truly innovate and lead the industry forward, we recognized the need to free ourselves from legacy frameworks and embrace the collective intelligence of the community. This realization led us to an important decision: it was time for Pyrodactyl to become a community-driven project.

We believe that by handing over the reins to the community, we can foster an environment where creativity and innovation thrive without limitations. This transition represents a significant step in our broader strategy to push the boundaries of what’s possible in hosting.

In conjunction with relicensing, we are transitioning Pyrodactyl’s development to a community-driven model. We have granted multiple community members maintainer access to the Pyrodactyl repository, giving them the authority to steer the project in new directions and ensure its continued growth and success.

This transition reflects our broader vision for the future of hosting. We see this as the final frontier—a space where possibilities are endless, and the only limits are those we impose on ourselves. By opening up Pyrodactyl to the community, we are breaking down those limits and inviting everyone to contribute to the evolution of hosting technology.

Looking Ahead

As we embark on this new chapter for Pyro, we are filled with excitement and anticipation for what the future holds. Our commitment to open-source principles remains as strong as ever, and we are already hard at work on groundbreaking projects that will redefine the hosting industry.

We're not just refining what exists – we're completely reimagining the future of hosting. Our team is developing a revolutionary new platform built from the ground up, incorporating cutting-edge technologies and a fresh, intuitive design. This isn't just an improvement; it's a paradigm shift in how hosting services are delivered and managed.

Let us be clear: Pyro is here to stay. We're not going anywhere. In fact, we're doubling down on our commitment to the hosting industry and our community. The transition of Pyrodactyl to a community-driven project is just the beginning. It lays the foundation for our next phase of growth and innovation.

We invite you to join us on this exciting journey. Whether you're a developer, a company, or an enthusiast, there's a place for you in our community. Together, we'll push the boundaries of what's possible in hosting, creating solutions that are more powerful, more accessible, and more aligned with the needs of users and businesses alike. Thank you for being a part of this chapter in our story. The best is yet to come, and we can't wait to shape the future of hosting with you by our side.