Object Storage in location HEL1 partially degraded

Zeitverlauf

1. 12:00 UTC Statusseite von Hetzner · Reported Object Storage in location HEL1 partially degraded
2. 08:23 UTC Statusseite von Hetzner · Läuft The situation for affected customers in hel1 has stabilized. Few requests may still take longer than usual to complete, and we are actively working on addressing that.
3. 11:34 UTC Statusseite von Hetzner · Läuft Unfortunately, the issue remains unresolved. Requests may still sometimes be delayed or time out for buckets located on the affected cluster. We have managed to narrow it down to certain low-level commit operations becoming stuck, sometimes for several seconds. Investigating this issue is difficult because these stalls never last long and affect different parts of the cluster at different times. We are continuing to investigate this issue. Thank you for your understanding.
4. 11:52 UTC Statusseite von Hetzner · Läuft We were able to use a workaround to contain the problem to such an extent that it no longer directly affects buckets; however, it is still not entirely clear how the situation occurred. We are still investigating the issue.
5. 14:55 UTC Statusseite von Hetzner · Wird beobachtet Our cluster is back online; we are continuing to monitor the situation
6. 14:33 UTC Statusseite von Hetzner · Wird beobachtet We are still experiencing occasional delays and are continuing to work on restoring the service in full.
7. 15:43 UTC Statusseite von Hetzner · Wird beobachtet Due to the increased utilisation of the Object Storage product, we are still experiencing partially elevated response times and stalled requests. We are implementing bucket migrations within HEL1 to address the I/O situation that has been causing performance issues for a subset of customers. These migrations will be performed in the background and will not require any changes on the customer side.
8. 13:34 UTC Statusseite von Hetzner · Behoben The situation has stabilised, so we are closing the incident