Write issues to data placement groups of type SSD in eu-se-1a (was: Slow storage due to traffic event in eu-se-1a)

Post mortem at 2022-11-02 09:30 AM (GMT+1, Europe/Stockholm timezone) an event caused a lot of storage traffic to occur in our eu-se-1a availability zone which affected availability in the form of slow storage.

Update (2022-11-03):

We’ve traced the root cause of this incident to an issue relating to a routine maintenance event where a small amount of the distributed data placement groups of type SSD could not be written.

This was due to our storage software having durability concerns because replica writes could not be guaranteed. The distributed data placement groups that could not be written in turn halts the I/O path for writes to that group, giving the appearance of resources not being available.

This started at 09:13:38, recovery was initiated at 09:25:34 and full service was restored at 09:28:22 (GMT+1 Europe/Stockholm)

We are very sorry for any inconvenience this might have caused our customers, we consider availability of our services to be paramount and will evaluate our routines and policies relating to the maintenance event in question.

Please note that the previous mentioned reason was a side-effect of the maintenance event and not the root-cause (or cause) for this incident.