Incident across several services

Zeitverlauf

1. 10:11 UTC Statusseite von GitHub · Reported Incident across several services
2. 11:38 UTC Statusseite von GitHub · Wird untersucht We are seeing recovery for Projects. Users may experience stale search results for Projects while indexing catches up.
3. 13:35 UTC Statusseite von GitHub · Wird untersucht Users may experience stale Project search results. We are working to increase indexing speed. All other services are available.
4. 17:01 UTC Statusseite von GitHub · Wird untersucht Updates to issue labels may be delayed in being reflected in Projects by about ~10 minutes. We have added some capacity to work through the backlog more quickly, but it'll likely be a few hours to complete processing the full backlog of messages. All other services are available.
5. 17:30 UTC Statusseite von GitHub · Wird untersucht We will post another update in approximately one hour to share our progress.
6. 18:42 UTC Statusseite von GitHub · Wird untersucht Continuing to investigate the lag that may be experienced in issue labels being accurately reflected in Projects. We are working on alternate solutions to process the backlog of label updates.
7. 20:26 UTC Statusseite von GitHub · Wird untersucht We are preparing to deploy a change that will mitigate the impact.
8. 21:39 UTC Statusseite von GitHub · Wird untersucht Updates to issue labels may be delayed in being reflected in Projects. We are continuing to deploy a change that will accelerate processing of the backlog of label updates. All other services are available. We will provide another update within the next hour.
9. 00:16 UTC Statusseite von GitHub · Wird untersucht We've deployed a change intended to accelerate processing of the backlog of issue label updates in Projects. A sizable backlog still remains and we continue working through it. All other services are operating normally. We will provide another update within the next hour.
10. 02:00 UTC Statusseite von GitHub · Wird untersucht We are continuing to process the backlog of issue label updates for Projects. Users may still see delays before label changes are reflected in Projects. All other services are operating normally.
11. 03:14 UTC Statusseite von GitHub · Wird untersucht We are continuing to process the backlog of issue label updates for Projects. Label changes may still be delayed. All other services are operating normally.
12. 04:55 UTC Statusseite von GitHub · Wird beobachtet The degradation has been mitigated. We are monitoring to ensure stability.
13. 04:55 UTC Statusseite von GitHub · Behoben Starting at 07:57 UTC on September 23, GitHub experienced elevated 500 and 404 responses across several application pages. This caused failures when installing GitHub Apps, creating organizations, and making some organization membership changes. Customers also experienced delayed label updates and stale search results in Projects. The infrastructure failure was isolated to our Azure Central US region. The API errors were mitigated by 10:58 UTC on September 23. Projects' processing continued to recover while an accumulated backlog was drained, and full service was restored at 04:55 UTC on…