Latest security news
Thu, October 22Yesterday
Increased latency processing integration webhooks
Oct 21 , 18:14 UTC Resolved - We are working on mitigating an incident that affects external integrations such as GitHub, Slack, and Autofix PRs. No data is being lost. Oct 21 , 17:53 UTC Investigating - We are working on mitigating an incident that affects external integrations such as GitHub, Slack, and Autofix PRs. No data is being lost.
Wed, October 21
Sentry.io not loading in US region
Oct 21 , 09:53 UTC Resolved - Native events ingestion has caught up, everything should be operational now Oct 21 , 09:40 UTC Update - Native events ingestion is still delayed, we are processing the backlog of native events Oct 21 , 09:23 UTC Monitoring - We have identified the problem, and addressed the cause of the outage. We will continue monitoring the fix Oct 21 , 08:54 UTC Investigating - We are currently investigating an issue affecting US Sentry region
Sentry.io not loading
Oct 20 , 21:38 UTC Resolved - We have a mitigation in place that should prevent this from occurring again in the short term while we continue to work on a proper fix. We are going to close the Statuspage incident for now, but will reopen it if things become unstable again. Oct 20 , 20:44 UTC Monitoring - We have put a better mitigation in place while we work on a fix. Things continue to be operational and we are monitoring the situation. Oct 20 , 19:51 UTC Update - The impact is mitigated and the dashboard/API are currently operational. We are continuing to work on a more durable fix. We will…
Tue, October 20
Sentry is down
Oct 19 , 21:32 UTC Resolved - Backlogs have been processed and we are back to normal. Oct 19 , 21:10 UTC Update - We are still working through our ingestion backlogs. At this point, it looks like we'll be caught up in the next 20 minutes. Oct 19 , 20:53 UTC Monitoring - We have identified and addressed the cause of the outage. Things are moving again and we are burning through ingestion backlogs. We will update the statuspage again as things progress. Oct 19 , 20:43 UTC Update - Impact is isolated to our US region customers. Alerting is also impact by this incident. Oct 19 , 20:39 UTC Investi…
Wed, October 14
Ingestion Halted for EAP data in EU
Oct 14 , 08:20 UTC Resolved - The issue with EAP ingestion in EU has been resolved. Oct 14 , 07:46 UTC Monitoring - In the EU we've identified an issue with ingestion into EAP and have implemented a fix. Waiting on ingestion lag burn down. Oct 14 , 06:09 UTC Update - In the EU we've identified an issue with ingestion into EAP and have implemented a fix. Waiting on ingestion burn down. Oct 14 , 05:46 UTC Identified - In the EU we've identified an issue with ingestion into EAP and are working towards a fix.
Wed, October 7
Sentry dashboard down
Oct 6 , 19:40 UTC Resolved - Dashboard is back to operational. Oct 6 , 19:20 UTC Monitoring - We have deployed a fix and are things are recovering. We are continuing to monitor the situation. Oct 6 , 19:04 UTC Identified - We have identified the issue and are working on a fix Oct 6 , 18:57 UTC Investigating - We are investigating an issue impacting our dashboard in both US and EU regions
Fri, September 18
Delayed ingestion of errors in EU
Sep 18 , 11:57 UTC Resolved - We are processing real-time data for all data types again. Sep 18 , 11:45 UTC Update - Error ingestion is fully operational. We are currently working on recovering real-time processing of spans. Sep 18 , 10:58 UTC Identified - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18 , 10:56 UTC Update - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18 , 09:54 UTC Monitoring - We are processing real-time data again and are working on burning our backlog. Sep 18 , 09…
Delays in spans ingestion
Sep 18 , 10:58 UTC Resolved - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18 , 10:56 UTC Update - We have identified a networking issue with an upstream provider and are working on mitigations. Sep 18 , 10:17 UTC Investigating - We are currently investigating an issue affecting spans ingestion in the EU region
Fri, September 11
Ingestion is delayed in US
Sep 10 , 17:10 UTC Resolved - Ingestion is now operating normally. Sep 10 , 16:52 UTC Update - We are continuing to monitor delayed ingestion. Sep 10 , 16:13 UTC Investigating - We are investigating degraded ingestion in US.
Sentry.io elevated number of 500 errors
Sep 10 , 16:26 UTC Resolved - Issue fully resolved Sep 10 , 16:07 UTC Monitoring - Issue were identified and fixes were applied. We continue to monitor situation Sep 10 , 15:25 UTC Identified - During migration to new control silo region we started to observe increased number of 500 errors. Migration procedure was rolled back.
Wed, September 2
Span ingestion is degraded in US
Sep 1 , 20:29 UTC Resolved - Spans ingestion is now operating normally. Sep 1 , 19:31 UTC Update - Latency is almost fully recovered. Sep 1 , 18:30 UTC Monitoring - A fix has been implemented, and latency is coming down. Sep 1 , 17:50 UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep 1 , 16:41 UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep 1 , 15:40 UTC Update - We are continuing to investigate an issue that affects spans ingestion latency Sep 1 , 14:16 UTC Investigating - We are experiencing slow i…
Tue, September 1
Sentry requests timing out / 504's
Sep 1 , 11:21 UTC Resolved - This incident has been resolved. Sep 1 , 09:03 UTC Update - We're monitoring the implemented fix Sep 1 , 08:57 UTC Monitoring - We identified the issue and rolled out a configuration change Sep 1 , 08:01 UTC Investigating - We are currently investigating what's going on and why requests timing out
Ingestion backlog
Aug 31 , 22:43 UTC Resolved - This incident has been resolved. Aug 31 , 21:26 UTC Update - Our systems are recovering from an ingestion backlog, webhooks are currently delayed Aug 31 , 21:16 UTC Monitoring - Our systems are recovering from an ingestion backlog Aug 31 , 21:12 UTC Investigating - API is experiencing issues Aug 31 , 21:02 UTC Monitoring - The fix has been deployed and our ingestion backlogs are recovering Aug 31 , 20:52 UTC Update - We have identified the culprit and are rolling out a fix Aug 31 , 20:31 UTC Investigating - we are investigating a failure in our infrastructure tha…
Fri, August 28
sentry.io is not available
Aug 27 , 15:41 UTC Resolved - Our latency is now back to normal. Incident resolved Aug 27 , 13:58 UTC Update - Our latency is now back to normal. We'll keep monitoring our systems a few more time before declaring the incident as resolved Aug 27 , 13:39 UTC Update - We keep monitoring our systems while we recover to our usual latency Aug 27 , 12:48 UTC Update - We are continuing to monitor while our systems recover after an issue that affected the availability of our product and API Aug 27 , 11:44 UTC Monitoring - We have identified and mitigated an issue that affected the availability of our…
Thu, August 27
Issues performance is degraded in the US
Aug 26 , 20:36 UTC Resolved - The issue has been resolved Aug 26 , 20:04 UTC Update - We have put a fix into place and are monitoring the situation Aug 26 , 18:47 UTC Monitoring - We have put a fix into place and are monitoring the situation Aug 26 , 17:53 UTC Identified - We have identified a potential cause of excess traffic and are working on a fix. Aug 26 , 16:17 UTC Investigating - We are experiencing slower page loads and API responses due to a database issue in us
Tue, August 25
Errors alerting degraded in US region
Aug 25 , 12:38 UTC Resolved - Error alerting is back to normal. Aug 25 , 12:27 UTC Identified - We have identified the issue and implemented a fix. Aug 25 , 11:09 UTC Update - We are continuing to investigate the issue Aug 25 , 09:38 UTC Investigating - We are investigating delays in error alerts in the US region, up to 10 minutes
Wed, August 19
Alert backlogs in US
Aug 18 , 22:21 UTC Resolved - The problem has been resolved, all backlogs have cleared and everything is operational again. Aug 18 , 22:15 UTC Monitoring - We have implemented a fix and the systems are recovering. We are monitoring to ensure that backlogs are proessed. Aug 18 , 21:56 UTC Identified - We have identified the cause and are working to determine a mitigation. This is only affecting US right now. Aug 18 , 21:42 UTC Investigating - We are currently investigating a delay in evaluating alerts in the US region.
Fri, August 14
Span & Transaction ingestion delayed in DE
Aug 13 , 22:27 UTC Resolved - All backlogs have been processed. Aug 13 , 22:13 UTC Update - We continue to make progress on the spans backlog with an estimated completion time of 15 minutes. Aug 13 , 21:22 UTC Update - We're continuing to make progress on the span ingestion backlog in US. Aug 13 , 20:32 UTC Update - We've recovered all ingestion latency in DE. In US, errors have processed their backlog and we continue to make progress on spans in US. Aug 13 , 19:53 UTC Update - The problematic change has been reverted and we're processing backlogs. Aug 13 , 19:34 UTC Monitoring - The problema…
Thu, August 13
Error ingestion delays in US
Aug 12 , 22:52 UTC Resolved - Backlogs have been processed Aug 12 , 22:43 UTC Identified - We've mitigated the source of ingestion latency and are processing the backlog. Aug 12 , 21:58 UTC Investigating - We are currently investigating error ingestion latency in US.
Wed, August 12
Delayed Error ingestion in our US region
Aug 12 , 14:48 UTC Resolved - The backlog has been processed Aug 12 , 14:21 UTC Monitoring - We're processing new events and alerts in realtime, and continue to process the tail of the backlog. Aug 12 , 13:28 UTC Identified - A fix has been made and the backlog is being processed. Aug 12 , 12:36 UTC Update - We've identified the root cause and are actively working on mitigation. Aug 12 , 11:45 UTC Investigating - We are actively investigating this issue.
Sentry API timeouts (US)
Aug 12 , 14:24 UTC Resolved - API performance has returned to expected levels. Aug 12 , 13:14 UTC Monitoring - API performance has returned to expected levels. We are continuing to monitor. Aug 12 , 12:58 UTC Investigating - We are currently investigating reports of API timeouts for the Sentry API in our US region.
Sat, August 8
Spans, crons, logs ingestion delayed in US region
Aug 7 , 17:08 UTC Resolved - The ingestion delay is resolved Aug 7 , 16:48 UTC Monitoring - Ingestion latency for spans, crons, and logs is back to normal levels. Aug 7 , 16:30 UTC Identified - We've put a fix in place and are monitoring. We anticipate catching up to ingestion within 1 hour Aug 7 , 16:21 UTC Investigating - We are currently investigating the issue