DEV Status - Incident History

Incident with DNS configuration

2022-04-06T16:41:55Z

Apr 6, 16:41 UTC
Resolved - This incident has been resolved. DNS appears to be propagated but may still not resolve for some users due to local DNS caching.

Apr 6, 16:13 UTC
Update - DNS propagation is close to complete.

Apr 6, 15:22 UTC
Monitoring - A fix has been implemented and we are awaiting DNS propagation.

Apr 6, 15:12 UTC
Identified - The issue has been identified and a fix is being implemented.

Apr 6, 14:58 UTC
Investigating - We are currently investigating a DNS issue.

Database Maintenance

2022-02-19T02:21:12Z

Feb 19, 02:21 UTC
Completed - The scheduled maintenance has been completed.

Feb 19, 01:05 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Feb 19, 01:03 UTC
Scheduled - DEV will be down for scheduled maintenance for one hour starting at 01:00 UTC (5PM PT/8PM ET/02:00 AM CET/6:30 AM IST) today.

Database Maintenance

2022-01-15T02:02:01Z

Jan 15, 02:02 UTC
Completed - The scheduled maintenance has been completed.

Jan 15, 01:03 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 15, 01:02 UTC
Scheduled - DEV will be down for scheduled maintenance for one hour starting at 01:00 UTC (5PM PT/8PM ET/02:00 AM CET/6:30 AM IST) today.

Website Performance Significantly Degraded

2020-10-19T17:57:05Z

Oct 19, 17:57 UTC
Resolved - We have killed the offending migration. The site should be back to operating normally now.

Oct 19, 17:37 UTC
Identified - We have identified an issue where the user table in our database has been locked due to a recent migration. We are figuring out how to handle the issue and will keep you posted as the situation evolves. Thank you for your patience.

Degraded App Performance

2020-08-13T20:32:08Z

Aug 13, 20:32 UTC
Resolved - This incident has been resolved.

Aug 13, 20:15 UTC
Monitoring - Database infrastructure upgrade is complete. Response times seem to be returning to normal, we will continue to monitor the situation to ensure the app continues to remain stable and performant.

Aug 13, 19:25 UTC
Update - We are continuing to work on upgrading our database. We will report back once the upgrade is complete or if anything changes with the status of the app.

Aug 13, 18:52 UTC
Update - We are currently in the process of upgrading our database to help us handle the additional load on the website. We will post another update in 30 minutes. Thank you for your patience as we work to resolve this issue.

Aug 13, 18:34 UTC
Identified - We have identified that the recent degraded performance is being caused by some database issues. We are working to resolve the problem. We will update in the next 15 min.

Postgres Database Maintenance

2020-08-09T01:26:49Z

Aug 9, 01:26 UTC
Completed - Migrations were a success, all operations should be back to normal. Thank you for your patience!!!

Aug 9, 00:58 UTC
Verifying - Database migrations have completed. We are rechecking all systems now

Aug 9, 00:51 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 9, 00:51 UTC
Scheduled - We are currently migrating the final table, notifications, which has had a much larger load on our servers than anticipated causing most pages to timeout. At this point we plan to let the migration finish. Will have another update in 15 min

Aug 9, 00:01 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 8, 18:26 UTC
Scheduled - During this time we will be running some large table migrations which will cause certain tables to lock. These table locks will cause degraded performance in portions of the application. The tables that are being updated are ahoy_messages, notifications, and articles.

Further technical information can be found in the pull request used to make these changes, https://github.com/forem/forem/pull/9476.

Incident on 2020-07-30 15:44:11 UTC

2020-07-30T18:54:59Z

Jul 30, 18:54 UTC
Resolved - This incident has been resolved.

Jul 30, 16:27 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Jul 30, 15:46 UTC
Identified - We've investigated and identified the cause of missing profile images and the fix is being applied.