Packit Service Status

[Resolved] Testing Farm tests incorrectly cancelled

Mon, 02 Mar 2026 20:00:00 +0100

A bug introduced in the new Packit service deployment caused a race condition that incorrectly cancells newly triggered running Testing Farm tests. We are deploying the fix.

Resolved: The fix has been deployed. Please make sure you didn’t merge PRs that have neutral status set by this bug and retrigger the builds/tests via /packit build or /packit test.

[Resolved] DB Cleanup

Sat, 14 Feb 2026 12:00:00 +0000

We are performing scheduled database cleanup operations this weekend (February 14-15). This maintenance should not impact users.

In case you experience any issues, please contact us. We will provide updates.

[Resolved] Packit outage

Thu, 05 Feb 2026 18:00:00 +0100

There was a Packit outage that prevented jobs from proceeding. As a result, we have a long-standing queue, and new jobs can take hours before being processed.

[Resolved] Copr outage - stuck hypervisor

Tue, 13 Jan 2026 02:00:00 +0100

There was a Copr outage and as a result Copr builds were stuck in a pending state.

Now the outage is over and any pending jobs should be processed automatically. If they aren’t trigger them manually.

However, there is a long queue of builds in Packit and it could take hours for it to decrease.

[Resolved] Christmas is here 🎄

Mon, 22 Dec 2025 08:00:00 +0100

From December 22nd, the team will be on vacation. We’ll be back starting January 5th, refreshed and ready to tackle new Packit features and bugs!

[Resolved] Testing Farm outage

Tue, 16 Dec 2025 13:00:00 +0000

Testing Farm had an outage caused by the provisioner going down. The outage has been addressed, you can see more details: https://status.testing-farm.io/issues/2025-12-16-public-ranch-outage/.

Testing jobs should now be able to be processed, but there is a backlog of testing requests in Testing Farm, which may take a few hours to be processed.

[Resolved] Testing Farm outage

Wed, 10 Dec 2025 15:30:00 +0000

Testing Farm had an outage caused by a recent change in Copr frontend, affecting most of Testing Farm runs in Packit. The Copr team has applied a fix to resolve the issue. More details: https://status.testing-farm.io/issues/2025-12-10-public-ranch-requests-testing-against-copr-builds-are-broken/.

If the jobs ended on error they can be retriggered by commenting /packit test in the corresponding pull requests.

[Resolved] Slow/delayed processing of Packit jobs

Wed, 03 Dec 2025 00:00:00 +0000

Packit Service was overflown with tasks (most likely caused by Copr having trouble starting builds), hit resource limits and as a result stopped processing tasks for about 9 hours. We’ve increased resources and got things running, but it will take a while before the accumulated queue of tasks is processed and things get back to normal.

Update: the queue has been processed, everything should be back to normal.

[Resolved] Copr outage

Wed, 26 Nov 2025 08:00:00 +0100

Copr services are temporarily unavailable due to service outage.

In Packit, this results in Copr builds stuck in a pending state.

[Resolved] Fedora Anubis

Thu, 20 Nov 2025 01:00:00 +0000

Packit downstream jobs may not work as expected due to the new Anubis rules deployed in Fedora Infrastructure. For example, sometimes Packit gets back an HTML page with the Anubis disclaimer instead of the specfile or the packit yaml config. Waiting and retriggering a job could work around the issue.

[Resolved] Fedora Infra - Datacenter Move outage

Mon, 30 Jun 2025 01:00:00 +0000

Fedora Infrastructure is currently ongoing migration to different datacenter, during this migration majority of the contributor-facing services will be down.

Therefore it is expected that sync-release (i.e., propose-downstream and pull-from-upstream), koji_build, and bodhi_update Packit workflows will be affected.

Copr build jobs may be disrupted, full outage is not expected.

For more details watch:

Update - 4th July

Most services in Fedora Infrastructure should be up and running again. Packit has resumed processing copr_build jobs, but there are still delays due to jobs being queued during the outage in previous days. We are continuing to investigate issues related to pushing and pulling from dist-git, which are currently impacting downstream jobs. We’ll share further updates as soon as we have more information.

[Resolved] Copr SRPM builds failing

Wed, 05 Mar 2025 08:00:00 +0100

SRPM builds in Copr have been failing, the reason is still unknown, please be patient. You can monitor #buildsys:fedoraproject.org on Matrix for updates.

Update: Copr SRPM builds are working again. You can retrigger failed jobs with /packit build or /packit rebuild-failed.

[Resolved] Maintenance outage

Mon, 03 Feb 2025 15:15:00 +0100

Today’s weekly production deployment requires a smaller outage to ensure proper redeployment without any inconsistencies.

We expect an outage for up to 15 minutes. There will be no jobs processed from 2025-02-03 15:15 UTC till the successful redeployment.

In case you are affected, please retrigger the jobs accordingly (via comments, or Re-Run button on GitHub Checks) once the outage is over.

Thanks for the understanding.

[Resolved] Longer response times

Mon, 06 Jan 2025 15:30:00 +0100

You may notice longer response times for your Packit jobs during certain intervals. We are trying to fix the root cause and apologize for the inconvenience!

[Resolved] Christmas is here 🎄

Tue, 24 Dec 2024 08:00:00 +0100

Starting from 24th of December all the team will be on vacation.

Please, give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs! We should be back on January 2nd.

Have a wonderful year 2025 everyone! We’re back.

[Resolved] Copr outage

Mon, 23 Dec 2024 06:00:00 +0100

Copr services are temporarily unavailable due to a service disruption.

In Packit, this results in Copr builds stuck in a pending state.

[Resolved] Copr outage - system update

Wed, 04 Dec 2024 10:00:00 +0100

There is a Copr outage and as a result Copr builds can be stuck in a pending state.

Update: the outage is over, any pending jobs should be processed automatically. In case some jobs failed to be triggered, you can trigger them manually.

[Resolved] OpenScanHub outage

Fri, 01 Nov 2024 08:00:00 +0100

OpenScanHub service is having an outage resulting in scans submitted by Packit being stuck in a FREE state.

You can monitor the progress of resolving the underlying issue here: https://pagure.io/fedora-infrastructure/issue/12160.

[Resolved] Maintenance outage

Wed, 09 Oct 2024 20:00:00 +0200

Database volume needs to be enlarged to accomodate the load on the Packit Service. To minimize the amount of jobs affected by this maintenance there will be no jobs processed, but they should be enqueued until the outage is over.

In case you are affected, please retrigger the jobs accordingly (via comment on the affected pull requests, or Re-Run button on GitHub Checks).

Thanks for understanding.

[Resolved] Testing Farm outage

Tue, 01 Oct 2024 10:00:00 +0200

Testing Farm infrastructure is having an outage resulting in test jobs being stuck in queued state. You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-10-01-public-ranch-outage/.

If the jobs end on error they can be retriggered by commenting /packit test in the corresponding pull requests.

[Resolved] Fedora infrastructure outage

Sun, 15 Sep 2024 08:00:00 +0000

Fedora infrastructure is having an outage resulting in some Copr build jobs failing with error Failed to download metadata... Librepo error: Cannot download repomd.xml... You can watch the status of the issue here: https://pagure.io/fedora-infrastructure/issue/12183.

[Resolved] Testing Farm outage

Fri, 30 Aug 2024 08:30:00 +0000

If the jobs end on error they can be retriggered by commenting /packit test in the corresponding pull requests.

[Resolved] Testing Farm outage

Wed, 28 Aug 2024 12:00:00 +0000

Testing Farm infrastructure is having an outage resulting in test jobs being stuck in queued state.

The affected test jobs should be retriggered by commenting /packit test in the corresponding pull requests.

[Resolved] Testing Farm outage

Mon, 05 Aug 2024 08:00:00 +0200

Testing Farm Fedora images are outdated and public ranch requests processing paused.

You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-08-05-expired-images-f39-f40/.

[Resolved] Testing Farm outage

Fri, 02 Aug 2024 06:00:00 +0200

Testing Farm public ranch is having an outage resulting in tests failing for fedora-rawhide-x86_64 target. You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-08-02-rawhide-outage/.

[Resolved] Testing Farm outage

Mon, 22 Jul 2024 00:16:00 +0000

Testing Farm infrastructure is having an outage resulting in tests stuck in a queued state. You can watch the status of the outage here: https://status.testing-farm.io/issues/2024-07-22-public-ranch-outage/

The tests should be executed once the outage is resolved.

[Resolved] release-monitoring.org outage

Wed, 26 Jun 2024 08:00:00 +0100

There is a New Hotness outage resulting in no Bugzillas about new releases being filed. As a result, no Packit update dist-git pull requests will be created.

There is a workaround: edit your release-monitoring.org project and tick the Check latest release on submit checkbox.

Update: It seems that the issue has been resolved and new release bugzillas are being filed automatically again.

[Resolved] Fedora integration outage

Mon, 27 May 2024 14:20:00 +0200

One of dependencies for our Fedora Messaging consumer has not been installed during the image build which resulted in an outage of the Fedora downstream integration.

Fedora integration is disrupted:

since 2pm CEST for our stage deployment
since 6pm CEST for our production deployment

Please trigger the respective jobs manually, if you are affected, once the outage has been resolved.

We are very sorry for the inconvenience!

Edit #1: Issue has been resolved for both production and staging deployments by rolling back to the previous image builds.

[Resolved] New Hotness outage

Thu, 09 May 2024 03:00:00 +0000

There was a New Hotness outage resulting in no Bugzillas about new release being filed. As a result, no Packit update dist-git pull requests were created.

Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:

/packit pull-from-upstream

We are very sorry for the inconvenience!

[Resolved] Outage

Tue, 07 May 2024 13:00:00 +0200

There has been a Packit outage. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!

[Resolved] Outage

Thu, 25 Apr 2024 14:30:00 +0200

There has been a Packit outage. We are investigating the reason of it. The jobs in the queue are now being picked up again and should be processed gradually (you might have experienced a delay in job execution). We are very sorry for the inconvenience!

[Resolved] Many Copr builds pending

Tue, 23 Apr 2024 14:00:00 +0100

The Copr backend is under a huge load and many builds are still in a pending state. The Copr builds will take more time than usual to finish. You can monitor the pending queue here: https://copr.fedorainfracloud.org/status/stats/.

[Resolved] GitHub outage

Fri, 05 Apr 2024 10:15:00 +0100

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com

[Resolved] Fedora infrastructure authentication problem

Mon, 26 Feb 2024 13:00:00 +0000

There is a problem with authentication in Fedora Infrastructure, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.

The problem should be now resolved. You can retrigger the failed jobs by:

/packit propose-downstream comment in any upstream issue
/packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request

[Resolved] Koji Outage

Wed, 21 Feb 2024 21:00:00 +0000

Planned Outage:

Koji upgrade at 2024-02-21 21:00 UTC

There will be an outage starting at 2024-02-21 21:00 UTC, which will last approximately 3 hours.

Reason for outage:

Koji will be upgraded to 1.34.0, which requires a schema update that touches many rows. Koji team estimate this will take about 45minutes to complete and during that time, Koji service will be completely offline.

You should avoid to trigger Koji builds in Packit (e.g. merging distgit Packit pull requests). You can retrigger the build, if it is missing, commenting a pull request with:

/packit koji-build

[Resolved] Pull From Upstream Outage

Wed, 21 Feb 2024 00:00:00 +0000

pull_from_upstream jobs were not executed properly at the given time frame because of an expired token.

Please trigger the release syncs manually if you are affected by commenting in any dist-git pull request:

/packit pull-from-upstream

We are very sorry for the inconvenience!

[Resolved] Fedora infrastructure authentication problem

Mon, 22 Jan 2024 22:00:00 +0100

The Fedora account system is currently down, leading to failures of jobs that interact with it (mostly propose_downstream, pull_from_upstream, koji_build, bodhi_update) as Packit is unable to authenticate. To stay updated on the root cause of this problem, please track this issue.

The problem should be now resolved. You can retrigger the failed jobs by:

/packit propose-downstream comment in any upstream issue
/packit pull-from-upstream / /packit koji-build / /packit create-update comment in any dist-git pull-request

[Resolved] Outage of automatic triggering of downstream Koji builds

Thu, 11 Jan 2024 09:30:00 +0100

Since 2024-01-11 until 2024-01-15 downstream Koji builds weren’t automatically triggered due to Fedora Infrastructure changes. Please trigger your builds manually if you are affected.

[Resolved] GitHub outage

Tue, 09 Jan 2024 14:45:00 +0100

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com

[Resolved] Christmas is here 🎄

Fri, 22 Dec 2023 19:00:00 +0100

All the team is on vacation. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.

–

Have a wonderful year 2024 everyone! We’re back.

[Resolved] GitLab outage

Tue, 19 Dec 2023 05:30:00 +0100

Packit production instance has lost access to GitLab API. This most likely affects also self-hosted GitLab forges.

The issue has been resolved and everything should work as expected.

[Resolved] Production instance firewall

Sun, 10 Dec 2023 20:45:00 +0100

We have moved our production instance to another OpenShift cluster which may result in some jobs failing. This issue should mainly affect jobs that involve syncing to the dist-git, namely:

propose-downstream
pull-from-upstream

In case you hit any issues when running propose-downstream or pull-from-upstream, please let us know.

Known Issues

none as of now (2023-12-11 17:00 UTC)

[Resolved] Testing Farm outage

Sun, 10 Dec 2023 20:40:00 +0100

Testing Farm provisioning is currently failing (due to failing DNS resolution). All runs created on the Testing Farm end with:

⚠ Guest couldn't be provisioned: Artemis resource ended in 'error' state

In case it affects you, please retrigger the tests via /packit test comment command in the pull requests, once the outage is resolved.

[Resolved] Production move

Sun, 10 Dec 2023 20:15:00 +0100

UPDATE #2: No issues have been noticed so far.

UPDATE: Production is deployed again. We are monitoring the deployment for any potential issues.

Our production deployment is being moved to a different cluster, you may experience outage for at most two hours.

In case you experience any issues, please retrigger the jobs manually via comment.

[Resolved] Copr builds stuck in pending or running state

Wed, 06 Dec 2023 12:00:00 +0100

Copr builds could have been stuck in a pending or running state. The issue has been resolved, it may take a while for the affected Copr builds to finish, but there is no need to retrigger them.

[Resolved] Copr Upgrade

Tue, 28 Nov 2023 13:30:00 +0000

Due to an upgrade of the Fedora Copr Servers, Copr builds might have been delayed or failing.

[Resolved] COPR builds failing for epel8

Tue, 14 Nov 2023 10:00:00 +0100

Building on COPR for epel8 chroot will be broken today for several hours. Today is release day of RHEL 8.9 and everyone is seeing 404 not found error when trying to access RHEL 8 repositories.

[Resolved] Production move

Sun, 12 Nov 2023 15:00:00 +0100

Our production deployment is being moved to different cluster, you may experience outage for at most an hour.

In case you experience any issues, please retrigger the jobs manually via comment.

UPDATE: Migration has been reverted due to issues with Fedora infra (authentication & message bus).

[Resolved] GitHub outage

Tue, 07 Nov 2023 14:45:00 +0100

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com

[Resolved] GitLab.com outage

Tue, 10 Oct 2023 04:00:00 +0200

GitLab.com workflows are disrupted due to the issue with detecting the upstream/downstream repository in the current working directory.

Self-hosted GitLab forges are not affected.

[Resolved] Cluster upgrade

Mon, 02 Oct 2023 08:00:00 +0000

A cluster upgrade is scheduled to start on Mon October 2nd at 08:00 UTC and last approximately 90 minutes.

We do not expect any significant disruption.

The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] COPR build failures

Sat, 09 Sep 2023 12:00:00 +0000

Copr is experiencing issues with builds. We are sorry about that. You can follow the progress in the Copr issue.

The failure in the COPR logs looks like the following: Machine 5b8689e424334255ab9ace8a58343076 still running. Killing...

In case you are building in your own COPR project you can go to your project settings and change Mock bootstrap from default to Enable for a quick workaround.

[Resolved] Copr frontend instability

Tue, 05 Sep 2023 15:15:00 +0200

Copr is experiencing issues with its frontend which is currently mitigated while Copr team investigates.

In case some of your builds were not submitted, please retrigger.

[Resolved] System upgrade scheduled

Mon, 28 Aug 2023 10:00:00 +0200

A system upgrade was scheduled and lasted approximately 90 minutes.

The started jobs might’ve been executed with a delay or dropped, if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] Testing Farm outage

Thu, 24 Aug 2023 15:40:00 +0200

Testing Farm infrastructure had a small outage that caused few failing test runs. In case it affected you, please retrigger the tests via /packit test comment command in the pull requests.

[Resolved] Packit outage now resolved

Mon, 21 Aug 2023 14:45:00 +0200

There has been a Packit outage now resolved. If your jobs are stuck, please retrigger them. We are sorry for the inconvenience.

[Resolved] Copr Upgrade

Wed, 16 Aug 2023 14:30:00 +0200

There will be a Copr outage starting at

date --date "2023-08-16 12:30 UTC"

which will last approximately 2 hours. The build queue processing will be stopped. During the outage time, we expect a very short downtime for the DNF package storage (less than 10 minutes).

Reason for outage:

We will update the infrastructure machines to the latest packages that are currently being developed.

Affected Services:

https://copr.fedorainfracloud.org/ https://download.copr.fedorainfracloud.org/results/

Upstream ticket:

https://github.com/fedora-copr/copr/issues/2824

Please join #fedora-buildsys, #fedora-admin or #fedora-noc on irc.libera.chat or add comments to this ticket.

[Resolved] Outage

Tue, 15 Aug 2023 04:00:00 +0200

We are having an outage that started on Tuesday EU early morning. Jobs during this time could get stuck in pending state. This is caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We are working on the fix and apologize for the inconvenience.

Update: Migration is completed. You might need to retrigger the jobs (i.e. via comment).

[Resolved] Staging instance firewall

Mon, 24 Jul 2023 14:30:00 +0200

We have moved our staging instance to another OpenShift cluster which may result in some jobs failing. This issue should mainly affect jobs that involve syncing to the dist-git, namely:

propose-downstream
pull-from-upstream

In case you are using our staging instance and you hit any issues when running propose-downstream or pull-from-upstream, please let us know.

Known Issues

~~Testing Farm is currently not available on the staging instance.~~ Edit: This issue has been fixed as of July 26th 10am.

[Resolved] Testing Farm testing blocked

Wed, 19 Jul 2023 08:00:00 +0000

Testing via Testing Farm is blocked: https://status.testing-farm.io/issues/2023-07-19-public-ranch-outage/.

Testing Farm team is working on the fix (ETA 2 hours).

[Resolved] Guest couldn't be provisioned: Artemis resource ended in 'error' state

Mon, 03 Jul 2023 08:00:00 +0200

Some jobs submitted by Packit to the Testing Farm are pending with this error: "Guest couldn't be provisioned: Artemis resource ended in 'error' state".

Testing Farm has already resolved the issue; please re-submit the job with a comment like /packit test (more info on retriggering here) or through the GitHub checks button.

[Resolved] Fedora Rawhide Testing Blocked on dnf5

Thu, 22 Jun 2023 09:30:00 +0200

dnf has been upgraded to dnf5 in Rawhide, resulting in various issues when running tests in Testing Farm.

The Testing Farm team together with dnf5 team is working on fixing the issues and trying to resolve the situation ASAP.

If you want to help, here are upstream issues related to this problem:

[Resolved] Upgrade of Copr servers

Thu, 25 May 2023 10:00:00 +0000

Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.

[Resolved] GitHub outage

Thu, 11 May 2023 15:30:00 +0200

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com

EDIT: GitHub’s post mortem

[Resolved] Testing Farm errors

Wed, 10 May 2023 14:00:00 +0200

After the deployment of the new version of Testing Farm, you may have encountered Testing Farm runs that fail with the status Failed to checkout ref {sha}. Together with the Testing Farm team, we are working on a fix.

[Resolved] GitHub outage

Tue, 09 May 2023 13:50:00 +0200

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com

[Resolved] Bodhi Updates not created

Thu, 04 May 2023 11:30:00 +0200

We’re experiencing issues with creating Bodhi updates due to failing authentication.

You can try to retrigger an update by a comment in a dist-git pull request:

/packit create-update

[Resolved] Cluster Update

Wed, 03 May 2023 10:00:00 +0200

The OpenShift cluster is currently updating from 4.11.28 to 4.12.13

[Resolved] Scheduled DB Cleanup

Wed, 26 Apr 2023 04:40:00 +0000

We are running a DB maintenance in order to clean up old pipelines. We expect API performance to improve as a result of this.

The maintenance window is planned for an hour, the actual clean up activities will last approximately 20 minutes. Users might experience degraded performance of the service and dashboard during this time.

[Resolved] Issue with pushing to dist-git

Tue, 11 Apr 2023 18:28:00 +0200

Due to a new pre-push hook being applied in dist-git, we are unable to create pull-requests from fork as part of propose-downstream and pull-from-upstream jobs.

We are sorry about that and are working on a way how to resolve this.

[Resolved] Upgrade of Copr servers

Wed, 05 Apr 2023 12:00:00 +0000

Due to an upgrade of the Fedora Copr Servers, Copr builds might be delayed or failing. The upgrade should last approximately 3 hours.

[Resolved] SLO1 Error Budget issues

Thu, 09 Mar 2023 10:00:00 +0100

We are having SLO1 error budget issues.

These are mostly caused by performance issues, we are working on them and prioritizing more improvements (see https://github.com/packit/packit-service/issues/1954 ).

We are sorry for any slowness you can experience.

[Resolved] Running grouping migrations

Tue, 14 Feb 2023 02:00:00 +0100

We are updating our database schema to facilitate further development. As a part of this process, multiple database migrations will have to be run. The first step is grouping together related builds and test runs. The migrations will happen during the next deployment to production, on Tuesday February 14 at 2:00AM CET. This set of migrations is expected to take roughly 40 minutes, after which the service should come back up.

Since significant refactoring was paired with these changes, there may be some bugs/regressions that we did not spot on staging. Please, contact us via the usual means if you spot anything suspicious.

[Resolved] SLO2 Error Budget Depleted in a Day

Tue, 31 Jan 2023 10:00:00 +0100

The SLO2 error budget dropped from 85% all the way down to zero in one day.

The likely cause was that the fedora-messaging consumer was stuck, so we were not getting messages from Copr about finished SRPM builds. Luckily, the babysit tasks proved to be very useful and caught those later.

We’ve added a liveness probe to the messaging consumer to prevent this from happening again.

[Resolved] Testing Farm outage

Tue, 24 Jan 2023 15:00:00 +0100

Testing Farm infrastructure had an outage which caused the status of the test runs to be stuck in the queued state. This is now resolved and tests can be retriggered by /packit test comment command in the pull requests.

[Resolved] SRPM builds stuck in Submitted state

Fri, 20 Jan 2023 21:19:06 +0100

Workers for processing long-running tasks got stuck on Friday evening with to most visible consequence being the SRPM builds stuck in ‘Submitted’ state.

The workers were restarted on Sat morning and all SRPMs were sent to Copr to build and the jobs pipelines were processed.

The cause is not known at the time of resolving the issue.

[Resolved] System upgrade scheduled

Fri, 06 Jan 2023 10:00:00 +0000

A system upgrade is scheduled to start on Fri Jan 06 20:00 UTC and last approximately 90 minutes.

We do not expect any significant disruption.

The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] Christmas is here 🎄

Fri, 23 Dec 2022 18:00:00 +0000

All the team is on vacation. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.

The Packit staging instance (https://dashboard.stg.packit.dev/) will not run during this week. It will be back on the 2nd of January.

–

Have a wonderful year 2023 everyone! We’re back, staging instance is up and serving.

[Resolved] Upgrade of Copr Servers

Mon, 28 Nov 2022 16:00:00 +0100

Due to an upgrade of the Fedora Copr Servers, Copr builds might have been failing.

[Resolved] Staging is unstable

Fri, 25 Nov 2022 08:35:00 +0100

Since the last cluster update, we’ve been running our staging instance on a separate node pool which uses AWS Spot instances. It causes the staging instance to be quite unreliable. We’re testing it in order to find balance between price and usefulness.

[Resolved] Outage

Thu, 24 Nov 2022 23:24:00 +0100

We are having resource issues in our cluster and particular parts of the deployment can be down.

Updates:

2022-11-25T08:35:00+01:00: Stage instance has been scaled down so our production instance has more resources. Production instance should be fine now. We are investigating more.

[Resolved] System upgrade scheduled

Thu, 24 Nov 2022 19:00:00 +0000

A system upgrade is scheduled to start on Thu Nov 24 21:00 UTC and last approximately 90 minutes.

We do not expect any significant disruption.

The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] Copr builds and tests on commits and releases ignored

Tue, 01 Nov 2022 03:00:00 +0100

There was a bug introduced with Tuesday’s production update causing Copr builds and tests for commit and release triggers were not executed.

[Resolved] Copr storage move

Fri, 28 Oct 2022 21:00:00 +0200

Copr team is performing a backend storage move over the weekend of Oct 28-30th. Building in Copr will be disrupted or unavailable during this period.

For more info, please follow the [email protected]/thread/TRNW5JVCFIQJDUCND4KZWZVM7XVCYLVL/">Copr outage announcement.

Update: the upgrade was successful and Copr is fully operational.

[Resolved] System upgrade scheduled

Thu, 13 Oct 2022 19:00:00 +0000

A system upgrade is scheduled to start on Wed Oct 13 19:00 UTC and last approximately 90 minutes.

We do not expect any significant disruption.

The started jobs might be executed with a delay and if you think Packit does not react to your events, try to retrigger the action via comment command.

[Resolved] Issues with the task execution

Tue, 11 Oct 2022 13:57:00 +0200

We are experiencing issues with the execution of the tasks. (The statuses are waiting in the The task was accepted. state.) We are investigating the cause. Sorry about the possible troubles. We will try to fix this as soon as possible.

Updates:

Tue Oct 11 12:24:02 UTC: We are slowly going through the queue. Please, be patient.
Tue Oct 11 13:12:36 UTC: We have gone through the queue and you should experience regular response times. The original cause is still not known but the problem probably occured during an SRPM upload to Copr. (It can be a fault on our or Copr side.)
Wed Oct 12 05:27:46 UTC: Service operated normally since the last update. To reduce the chance of this, we plan to move the building of SRPMs to Copr. (Currently default for new installations and opt in for other projects.)

[Resolved] Internal Testing Farm infrastructure is having an outage

Mon, 10 Oct 2022 15:14:00 +0200

Due to a wide outage in Red Hat internal infrastructure the Red Hat Ranch (=internal ranch) is disabled in the Testing Farm (=our test runner). Currently there is no ETA for resolving the problem. Follow the progress on the Testing Farm status page: https://status.testing-farm.io/issues/2022-10-10-redhat-ranch/

[Resolved] GitHub Webhooks not working

Wed, 05 Oct 2022 16:00:00 +0200

Packit is not working on GitHub because the GitHub Webhooks status is ‘degraded’. See yourself: https://www.githubstatus.com/incidents/smn1qtqvbsb6

[Resolved] September 22th outage

Thu, 22 Sep 2022 08:00:00 +0200

Testing Farm is failing to run tests submitted by Packit which use Copr builds. Tests with the skip_build flag are not be affected by this issue. We are working together with the Testing Farm team to fix this as soon as possible.

[Resolved] 3/4 of SLO1 Error Budget consumed in 6 weeks

Thu, 01 Sep 2022 01:00:00 +0200

(Mostly for our internal use since the source data are not publicly available.)

During the 6 weeks since the beginning of September the SLO1 error budget dropped to 25%. After that (in the middle of October) the trend turned and now, at the end of October, it’s at 50%.

When looking at metrics in our (not public) Grafana (Packit boards -> (Prod/Stg) Accepted status time) we can see that the average value (of the “accepted status time”) indeed increased during the beginning of September by approx 1 second and cases of >15s started to appear so the error budget started to be consumed.

The cause is yet unknown, but the changes we did the last week of August and which could thus contribute to the problem were:

lots of SQLAlchemy-related changes - contribution unknown
switched 4 single-threaded workers to 1 multithreaded - there might be some correlation, but it’s not clear because for example the first 2 weeks of October the change was reverted (i.e. there were again the 4 single-threaded workers) and even the average value seemed to drop a bit, the error budget continued to be consumed (meaning there were still some cases of >15s).

We need to continue watching the metrics, experiment with the workers (numbers and types) and give the changes more time (2 weeks at least) to be able to tell whether they change the trend.

[Resolved] Failing SRPM builds done in Copr

Tue, 23 Aug 2022 04:00:00 +0200

SRPM builds happening in Copr (for packages using the srpm_build_deps config key) were failing due to a dependency issue.

[Resolved] Summer is here ☀️

Mon, 04 Jul 2022 08:00:00 +0000

Most of the team is on vacation. We try our best to solve issues with limited resources. Give us some time to get refreshed and be better prepared to work hard on Packit’s new features and bugs.

[Resolved] Upgrade of Copr Servers

Wed, 22 Jun 2022 14:00:00 +0200

Due to an ongoing upgrade of the Fedora Copr Servers, building might be delayed or failing. The upgrade should last ~3 hours.

[Resolved] SRPM builds done in Copr are failing

Tue, 21 Jun 2022 10:18:22 +0200

SRPM builds happening in Copr (for packages using the srpm_build_deps config key) are failing due to a dependency issue. We are working on a fix.

[Resolved] Networking Issue

Thu, 03 Feb 2022 08:37:21 +0100

Our production instance is getting various network issues. In logs, you can see errors with rsync, kerberos and other network related commands. We are sorry about the current state and are investigating the cause. We will let you informed once we know more.

edit: The problem was with one of our task runners. Most probably because of a memory issue we are now trying to find and resolve. The runner was recreated and is working fine currently. We are getting an eye on it; please let us know if you find anything suspicious.

[Resolved] February 1st outage

Tue, 01 Feb 2022 02:00:00 +0000

We had an outage on Tuesday EU night/early morning. Jobs during this time could get stuck in pending state. This was caused by a failed database schema migration because of a lack of memory during the automatic deployment of the new production images. We apologize for the inconvenience.

[Resolved] New production deployment

Wed, 08 Dec 2021 02:00:00 +0000

We’ve deployed production instance of Packit Service to a new Openshift cluster. If you spot anything unusual, let us know.

[Resolved] Longer response times

Tue, 07 Dec 2021 09:06:14 +0100

Our service got overwhelmed by the events coming from a Fedora dist-git (caused by the epel-9 branching) causing the longer responses.

We have scaled up the workers and the queue is getting shorter. We are planning the next steps to be resilient next time. Sorry about the disturbance.

update: The queue is now empty and you can expect regular operational times.

[Resolved] Problems with production database

Thu, 18 Nov 2021 02:33:00 +0000

After moving our production service to a new cluster, we’re seeing intermittent issues with connecting to the database. They are mostly manifesting in the dashboard. We need to decide next steps.

[Resolved] Moving production Packit Service to a new cluster

Thu, 18 Nov 2021 01:00:00 +0000

We are going to move deployment of our production Packit Service to a new cluster. It should take about 1/2 an hour, during which the workers will be down and queries to API & dashboard won’t work properly because the database will be being migrated. Any requests submitted to the API meanwhile should, however, be fulfilled once the workers are up again.

[Resolved] Slow SRPM builds

Tue, 07 Sep 2021 20:30:00 +0200

Building of SRPMs for specific projects needs more memory and blocks the worker for a longer time, this causes other jobs to be slow and delayed. We are working to resolve this in the week of September 13th.

[Resolved] August 27th outage

Fri, 27 Aug 2021 15:00:00 +0200

We had an outage which started in the Friday EU afternoon and was fully resolved on Saturday EU morning. Most of the jobs were failing during this time. This was caused by a misconfiguration on our side. We apologize for the inconvenience. We’re going to take steps to prevent the misconfiguration from happening in future.

[Resolved] Summer is here ☀️ #2

Thu, 01 Jul 2021 10:55:00 +0200

There are 2 days of public holidays on July 5th and 6th in Czech Republic and since most of the team is located in CZ, we’ll be unavailable during these two days.

[Resolved] Summer is here ☀️

Fri, 25 Jun 2021 10:55:00 +0200

Most of the team will be unavailable on Monday, June 28th. We’ll try our best to resolve issues as soon as we’re back.

[Resolved] PostgreSQL db upgrade

Fri, 04 Jun 2021 07:00:00 +0000

Upgrading our PostgreSQL database from version 10 to version 12.

[Resolved] Dashboard - unable to connect

Wed, 02 Jun 2021 11:16:00 +0000

API was down and therefore Dashboard was not showing any Jobs and after some time a “unable to connect …” error was shown.

Cause: Unknown

Fix: After restarting the API, the Dashboard started to work OK.

[Resolved] GitHub outage

Mon, 01 Jan 0001 00:00:00 +0000

There is an ongoing GitHub outage. You can monitor the status and subscribe for updates here: https://www.githubstatus.com