The homeserver suffered from an outage on Monday 22 April, between 08:00 UTC and 10:00 UTC, followed by slowness for the next 2 hours.

The outage occurred during scheduled maintenance with no expected downtime. It included upgrading the base OS of the machines running’s deployment.

Technical cause

The homeserver is large. To operate at this scale and distribute the load on several machines, it relies on Synapse workers.

The first machines upgraded only ran stateless workers and were trivial to upgrade. Upgrading the machines with stateful workers was more difficult. To avoid downtime, the stateful workers had to be moved to other machines from the same deployment, and be restarted.

The machines that received the additional workers ended up being resource-starved. In the meantime, it took significant time to upgrade the free machine and bring its network back online before we could move the workers back to it to better share the load.

Organizational cause is a historical and complex deployment that is difficult and costly to replicate entirely in a staging environment at this scale. It is also the largest deployment in the federation and has unique performance requirements.

Through this upgrade process, gaps in the deployment documentation were identified and these gaps contributed to the upgrade taking longer than expected. The deployment specificity has now been properly documented, and we are confident that the next machines’ upgrade will be significantly faster.

Lessons learned

To perform an upgrade without downtime, the Foundation would need to add another server in the deployment of the homeserver, as a consequence of steadily increasing traffic. An extra server introduces a recurring cost and requires additional work that the Foundation cannot afford right now.

The next upgrade will have about an hour of planned downtime and will be announced shortly. In the meantime, the Foundation is investigating the most cost-effective way to improve redundancy on

The Foundation needs you

The Foundation is a non-profit and only relies on donations to operate. Its core mission is to maintain the Matrix Specification, but it does much more than that.

It maintains the homeserver and hosts several bridges for free. It fights for our collective rights to digital privacy and dignity.

Support us