The overall error rate has remained stable since the larger incidents last week had caused big spikes again until initial infrastructure improvements were made to help mitigate:
The smaller spikes we saw yesterday and today around noon/afternoon line up with when we still saw increased saturation on our Sidekiq workers. (As well as the even flatter curve on the weekend aligning with less saturation due to generally lower load.) We're engaging with the infra team to understand the timeline for further improvements – one idea being explored was to separate out the CI-related Sidekiq workloads entirely to improve the resiliency. But while this may sound like an obvious solution, there is complexities involved that may count against that approach.
On the application side itself, we're looking into more ways to use caching or pure optimizations to further reduce the amount of I/O operations during config processing, in order to decrease the likelihood of running into the contention issues in the first place, see !227483 for the most current efforts.
Component and project includes are the most common usage patterns we see for the namespaces that are most affected. We've been mostly optimizing for component includes in our application-level efforts recently, and will begin looking into project includes next, based on what we've learned.
Finally, the increase of the GITLAB_CI_CONFIG_FETCH_TIMEOUT_SECONDS value has not actually taken effect yet – environment variables are a bit of an outdated pattern for adjusting something like this on GitLab.com, this one was mostly added to allow self-managed users to tune this value. I expect the increase to take effect within the next 24 hours. Hopefully Sidekiq remains stable as through the last few days, so we can more easily gauge the effect of the change. If it does cause a notable drop in error frequency and no other adverse effects, we can consider temporarily raising it even further.