Degraded snapshot upload latency

Zeitverlauf

1. 11:05 UTC Statusseite von Railway · Wird untersucht We’re investigating elevated latency on build snapshot uploads in EU West (Amsterdam). Builds in this region may take longer than usual to start. Running deployments are not affected.
2. 11:25 UTC Statusseite von Railway · Reported Degraded snapshot upload latency
3. 11:30 UTC Statusseite von Railway · Ursache gefunden We've identified the cause with our upstream storage provider. We're working with them to scale it up and restore normal upload times. Builds in EU West may still be slower than usual in the meantime.
4. 12:53 UTC Statusseite von Railway · Ursache gefunden Due to sustained elevated latency, some builds in EU West may fail early. We’re still working with our provider to scale storage capacity.
5. 15:06 UTC Statusseite von Railway · Wird beobachtet We’re seeing early signs of recovery in EU West. Snapshot upload times are improving and we’re monitoring closely.
6. 16:17 UTC Statusseite von Railway · Ursache gefunden We're seeing elevated snapshot upload latency in US West (California) and Southeast Asia (Singapore), caused by the same issue with our upstream storage provider. Builds in these regions may take longer than usual to start. We're working with our provider to restore normal upload times. EU West continues to recover. Running deployments are not affected. Note: This is an issue with our upstream storage provider, unrelated to the security incident from October 21st, which has been resolved. No action is needed on your part.
7. 19:14 UTC Statusseite von Railway · Ursache gefunden Ongoing
8. 20:29 UTC Statusseite von Railway · Ursache gefunden We're still seeing elevated snapshot upload latency in US West (California), caused by the same issue with our upstream storage provider. Builds in this region may take longer than usual to start. We're working with our storage provider to restore normal upload times. EU West and Southeast Asia are now in recovery. Running deployments are not affected. Note: This is an issue with our upstream storage provider, unrelated to the security incident from October 21st, which has been resolved. No action is needed on your part.
9. 22:57 UTC Statusseite von Railway · Wird beobachtet This latency is now normalizing in US West also, all other regions should be fully back to normal. We continue to monitor.
10. 23:43 UTC Statusseite von Railway · Behoben Snapshot upload latency has now resolved across all regions.