Logit.io Status - Incident History

Connectivity issues within our US Region

Tue, 21 Apr 2026 19:19:08 +0000

Apr 21, 19:19 UTC
Resolved - This incident has been resolved.

Apr 21, 16:46 UTC
Monitoring - We are seeing active recovery, we are monitoring

Apr 21, 16:37 UTC
Update - We have a restored access, we continue to work on brining back full availability

Apr 21, 16:22 UTC
Update - We are rebuilding some infrastructure while our service provider continues to fix underlying issues

Apr 21, 15:47 UTC
Identified - We are working with our Service provider to resolve.

https://status.us.ovhcloud.com/pages/incident/59dd23da8827c804746f1664/69e7946ca4603705eca31b72

Apr 21, 15:45 UTC
Update - We are continuing to investigate this issue.

Apr 21, 15:45 UTC
Investigating - We are currently investigating this issue.

Cloudflare Networking Issue

Tue, 18 Nov 2025 15:20:29 +0000

Nov 18, 15:20 UTC
Resolved - We are no longer seeing any impacts and have reverted our mitigations.

Nov 18, 15:06 UTC
Monitoring - Cloudflare has reported they have resolved the issue, we can access our dashboard.

We are keeping mitigations in place, and will slowly roll them back

Nov 18, 14:32 UTC
Update - We have updated the DNS entries of our Visualisation proxy so it can access our dashboard, and authenticate, you should be able to access.

Nov 18, 14:04 UTC
Update - We are actively working on mitigation the Cloudflare outage to our Dashboard.

Some of our caching is failing and cannot update due to the cloudflare issue, this may cause issues with accessing the visualisers

Nov 18, 13:09 UTC
Update - We are still seeing intermittent connectivity due to the Cloudflare issue, we are monitoring

Nov 18, 12:16 UTC
Update - We are seeing intermittent connectivity to our Dashboard

Nov 18, 12:00 UTC
Identified - We are aware of an issue effecting our Dashboard access and identified its an issue with Cloudflare Networking. All other components are not effected.

Status from Cloudflare:

Investigating - Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.

https://www.cloudflarestatus.com/

Resolved: US Visualization Hosts Intermittent Outage

Mon, 10 Nov 2025 14:00:00 +0000

Nov 10, 14:00 UTC
Resolved - Between 13:10 UTC and 13:57 UTC on November 10, our US visualization services (Kibana, Grafana, Jaeger) experienced intermittent connectivity issues due to instability within our Kubernetes cluster.

Total outage time: approximately 23 minutes and 20 seconds.

The issue was resolved by performing an ETCD backup restoration, and all services have remained fully operational since 13:57 UTC.

This incident was isolated to the US region, and no other services were impacted.

Database Switch for our Dashboard to improve performance and availability

Tue, 11 Mar 2025 17:29:03 +0000

Mar 11, 17:29 UTC
Completed - The scheduled maintenance has been completed.

Mar 11, 17:15 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 10, 11:18 UTC
Scheduled - There may be intermittent connectivity to our dashboard: dashboard.logit.io during the switch, all other services unaffected

Some Stacks not appearing on the Dashboard

Tue, 11 Mar 2025 16:30:00 +0000

Mar 11, 16:30 UTC
Resolved - Some customers are reporting that stacks do not appear on their dashboard.

This is not related to the upcoming maintenance, we are investigating.

We rolled back the latest build which resolved the issue

Rebooting Logs servers to apply updates

Mon, 03 Feb 2025 17:30:10 +0000

Feb 3, 17:30 UTC
Completed - All servers have been updated

Feb 3, 15:42 UTC
Update - UK region has been completed

Feb 3, 15:11 UTC
Update - The planned works have been completed in the US region

Feb 3, 10:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 30, 14:03 UTC
Scheduled - We will be rebooting some Logs search hosts to apply updates and change some disk mount options. Customers may notice reduced performance while their node starts up again but no customer outages are anticipated

Logstash Endpoint Certificate Rotation

Wed, 23 Oct 2024 15:00:03 +0000

Oct 23, 15:00 UTC
Completed - The scheduled maintenance has been completed.

Oct 21, 15:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Oct 21, 12:48 UTC
Scheduled - We will be rotating the SSL/TLS certificate used on your Logstash endpoints before the current certificate expires. This maintenance is necessary to ensure uninterrupted secure communication.

Important Notes:

No action is required on your part, as the intermediate and root certificates remain unchanged.
There will be no disruption to your service during this maintenance.

We appreciate your understanding and continued trust. Should you have any questions or concerns, please feel free to reach out to our support team.

OS updates for EU ingestion hosts (day3)

Fri, 31 May 2024 19:00:02 +0000

May 31, 19:00 UTC
Completed - The scheduled maintenance has been completed.

May 31, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 28, 16:13 UTC
Scheduled - We will be performing a rolling series of OS upgrades to the log and APM ingestion hosts.

This is an at risk notification, customers should not notice any service interruptions.

The Floating IPs will be pointed at their backup hosts during the maintenance window while the primary server is upgraded.

OS updates for EU ingestion hosts (day2)

Thu, 30 May 2024 15:42:02 +0000

May 30, 15:42 UTC
Completed - todays updated have been completed

May 30, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

OS updates for EU ingestion hosts (day1)

Wed, 29 May 2024 16:52:57 +0000

May 29, 16:52 UTC
Completed - Todays maintenance has been completed as planned

May 29, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 28, 16:11 UTC
Scheduled - We will be performing a rolling series of OS upgrades to the log and APM ingestion hosts.

This is an at risk notification, customers should not notice any service interruptions.

The Floating IPs will be pointed at their backup hosts during the maintenance window while the primary server is upgraded.

Alert servers OS upgrade

Wed, 08 May 2024 15:30:02 +0000

May 8, 15:30 UTC
Completed - Work on the EU alerting servers has been completed

May 8, 12:34 UTC
Update - Work on the UK alerting servers has been completed

May 8, 09:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 8, 08:58 UTC
Scheduled - We will be rebuilding the alert servers onto new hardware and with an updated OS.
There should be no break in alerting, there may be an issue with updating alert rules while an individual stack is migrating between alert hosts.

[Logs] [Europe] Filter server offline

Thu, 16 Nov 2023 05:30:41 +0000

Nov 16, 05:30 UTC
Resolved - The server is back online. All logstash filter containers have started up again.

Nov 16, 04:14 UTC
Update - We have started additional filter processes and almost all stacks should have processed their backlogs

The provider has reported a power issue and are expecting to provide their next update in 3 hours

Nov 16, 03:15 UTC
Monitoring - Our alerts have detected one of our filter servers has gone offline. No data should be lost, processing of incoming logs may be interrupted for some stacks.

SSL Certificate Rotation

Mon, 06 Nov 2023 18:00:40 +0000

Nov 6, 18:00 UTC
Completed - The scheduled maintenance has been completed.

Nov 6, 17:29 UTC
Update - Scheduled maintenance is still in progress. We will provide updates as necessary.

Nov 6, 11:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 6, 10:47 UTC
Scheduled - There should be no impact to customers

New Stacks and DNS Changes are failing due a cloudflare outage

Fri, 03 Nov 2023 08:13:49 +0000

Nov 3, 08:13 UTC
Resolved - This incident has been resolved.

Nov 3, 04:00 UTC
Update - We are continuing to monitor for any further issues.

Nov 3, 00:30 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Nov 2, 14:50 UTC
Identified - The issue has been identified and a fix is being implemented.

Cloudflare Pages and Load Balancers are offline

Mon, 30 Oct 2023 21:00:59 +0000

Oct 30, 21:00 UTC
Resolved - This incident has been resolved.

Oct 30, 20:29 UTC
Monitoring - It appears that cloudflare has resolved the issue for our Marketing site and Dashboard Authentication

Oct 30, 20:28 UTC
Update - Dashboard is operational for existing logged in users.

Our authentication provider has the same issue, and new users will not be able to Login

Oct 30, 20:10 UTC
Identified - Our upstream provider cloudflare has an outage, we will continue to monitor.

https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc

Data ingestion is still operational

Oct 30, 20:02 UTC
Investigating - We are currently investigating this issue.

Revert the dashboard mitigation from our provider outage last week, no impact expected

Mon, 07 Aug 2023 10:20:36 +0000

Aug 7, 10:20 UTC
Completed - The scheduled maintenance has been completed.

Aug 7, 10:15 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Aug 5, 09:00 UTC
Scheduled - We will be undergoing scheduled maintenance during this time.

Issues connecting to our dashboard

Fri, 04 Aug 2023 23:00:56 +0000

Aug 4, 23:00 UTC
Resolved - Our provider has resolved the issue

Aug 4, 15:46 UTC
Monitoring - We have applied some mitigation and you should be able to access our dashboard, we will continue to monitor our upstream provider

Aug 4, 15:37 UTC
Update - We are having issues with some load balancers due to our provider.

https://www.cloudflarestatus.com/incidents/cx141zv0rfd8

Aug 4, 15:35 UTC
Investigating - We are currently investigating this issue.

UK - Opensearch underlying hosts are offline

Wed, 10 May 2023 08:13:50 +0000

May 10, 08:13 UTC
Resolved - All stacks are operating as we expect and processing messages

May 10, 08:02 UTC
Update - Stacks are now processing queued messages

May 10, 08:00 UTC
Monitoring - Servers are back online, we are just monitoring the recovery

May 10, 07:56 UTC
Update - We should have the Rack back in less than 1 hour

May 10, 06:50 UTC
Identified - A rack at our data centre is offline effective a number of hosts, we are working with our provider to restore access. Update in 1 hour

May 10, 06:46 UTC
Investigating - We are currently investigating this issue.

Network Outage to Opensearch Dashboards and Kibana

Mon, 27 Mar 2023 22:57:00 +0000

Mar 27, 22:57 UTC
Resolved - We had a number of alerts for Opensearch Dashboard and Kibana instances that could not connect to the public network with our provider. We have confirmed that connections have been re-established and are working with our provider to understand the root cause of their outage.

We had connectivity issues between 22:57 to 23:22 UTC

Database Switch for our Dashboard to improve performance and availability

Mon, 20 Mar 2023 16:40:18 +0000

Mar 20, 16:40 UTC
Completed - The scheduled maintenance has been completed.

Mar 20, 16:35 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Mar 20, 16:32 UTC
Scheduled - There may be intermittent connectivity to our dashboard: dashboard.logit.io during the switch, all other services unaffected

SSL Certificate Maintenance

Mon, 21 Nov 2022 23:59:30 +0000

Nov 21, 23:59 UTC
Completed - The scheduled maintenance has been completed.

Nov 15, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 10, 10:33 UTC
Scheduled - We will be undergoing scheduled maintenance during this time, for more details see https://help.logit.io/en/articles/3306173-ssl-certificate-expiry-notice

DNS Nameserver Maintenance

Tue, 28 Jun 2022 17:06:33 +0000

Jun 28, 17:06 UTC
Completed - The scheduled maintenance has been completed.

Jun 28, 15:41 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jun 28, 15:38 UTC
Scheduled - We are migrating our DNS Nameservers. No impact is expected

A number of US Kibana and Grafana Hosts are offline

Fri, 17 Jun 2022 16:31:34 +0000

Jun 17, 16:31 UTC
Resolved - This incident has been resolved.

Jun 17, 15:02 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Jun 17, 14:11 UTC
Update - Our engineers are continuing to investigate this issue and will update in 1 hour.

Jun 17, 13:07 UTC
Update - Our engineers are continuing to investigate this issue and will update in 1 hour.

Jun 17, 13:05 UTC
Investigating - Our engineers are continuing to investigate this issue and will update in 1 hour.

Jun 17, 12:08 UTC
Update - We are continuing to monitor for any further issues.

Jun 17, 11:49 UTC
Update - We are continuing to monitor for any further issues.

Jun 17, 11:27 UTC
Monitoring - A fix has been implemented and we are monitoring the results.

Jun 17, 11:18 UTC
Update - Our engineers are continuing to investigate this issue and will update in 1 hour.

Jun 17, 10:36 UTC
Investigating - Our engineers are currently investigating this issue and will provide an update in 30 minutes.

Logit.io Response to Apache Log4j 2 CVE-2021-44228

Tue, 14 Dec 2021 00:23:42 +0000

Dec 14, 00:23 UTC
Resolved - All stacks have been patched and are operational. We have removed the org/apache/logging/log4j/core/lookup/JndiLookup.class file from all installations of log4j

Dec 13, 20:44 UTC
Update - All Elasticsearch clusters in the UK and US regions have been patched and updated. We continue in parallel to update the EU region

Dec 13, 20:05 UTC
Update - We can confirm we have applied the patches to all Logstash Inputs for Stacks in the EU Regions.

Dec 13, 16:59 UTC
Update - Update: We can confirm that we have now also applied the patches to Logstash for Filter nodes in the EU region.

Dec 13, 16:58 UTC
Update - Update: We can confirm that we have now also applied the patches to Logstash for Filter nodes in the EU region.

Dec 13, 14:10 UTC
Update - Update: Our engineers are in the process of rolling out a patched version of Logstash for both Input and Filter nodes for all active versions of ELK. We can confirm we have applied the patches to all Stacks in the UK and US Regions, and we are awaiting the completion of the EU Region.

Dec 13, 14:07 UTC
Update - Update: Our engineers are in the process of rolling out a patched version of Logstash for both Input and Filter nodes for all active versions of ELK.
We can confirm we have applied the patches to all Stacks in the UK and US Regions, and we are awaiting the completion of the EU Region.

Dec 13, 10:29 UTC
Identified - Follow updates here on how our teams are currently responding with the highest priority to vulnerability CVE-2021-44228 that is impacting multiple versions of the Apache Log4j 2 which was disclosed publicly via Github on December 9th 2021.

Logit.io engineers and security incident teams continue to actively analyse, identify and where necessary patch all affected log4j versions across all Logit.io data centre instances.

Logit.io are initially updating and following the current remediation advice as a first priority as detailed here https://github.com/advisories/GHSA-jfh8-c2jp-5v3q

"In previous releases (>=2.10) this behaviour can be mitigated by setting system property "log4j2.formatMsgNoLookups" to “true” or by removing the JndiLookup class from the classpath (example: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class)"

This remediation will be applied to all versions of Logit.io instances and we will post further updates here as this is rolled out.

In addition the Logit.io security team has analysed its logs and updated its monitoring of all internal services to include active alerting of any attempts to issue remote code execution via the JndiLookup class.

The information leak in Log4j does not permit access to data within an Elasticsearch cluster but allows an attacker to extract certain environmental data via DNS. The data leaked is limited to that available via Log4j lookups.

If you have any questions or concerns, please reach out to us via the normal support channels or email [email protected]. You can also choose to subscribe to updates at the top of this page to receive all future notifications.

Elasticsearch server offline

Tue, 23 Nov 2021 11:50:37 +0000

Nov 23, 11:50 UTC
Resolved - The server is back online - all stacks are showing green again

Nov 23, 11:37 UTC
Identified - One of our elasticsearch servers is offline. The issue has been raised with the provider and we will begin migrating the affected stacks onto other servers shortly.