Previous incidents

Queues have cleared up and are now maintaining a steady state.

Maintenance is continuing to progress successfully.

Completed:
- PostgreSQL database migration
- Gold-master VM snapshot
- Applied recent Glitch patches

Ran out of time steps:
- Partial Glitch patches
- Deploying ElasticSearch and completing an initial sync
- Deploying LibreTranslate and downloading required language libraries
- Reinstating monitoring products and initializing StatsD

Thank you for your patience as we worked through some of these steps. Another downtime may be ...

Web servers have been upgraded to a dedicated CPU cluster and a configuration patch has been implemented.

Our ISP's Engineering team has confirmed the resolution of the issue with networking in our data centre.

Infrastructure upgrades from Ubuntu 18.04 LTS to 22.04 LTS are underway.
Additional upgrades of Mastodon to the most recent glitch version are underway

Downtime is expected.

Please be advised that another round of infrastructure changes is scheduled for tonight and some downtime is expected.

Upgrades have been completed for the evening.

Unfortunately there are items that could not be actioned during this upgrade and are scheduled to the next available window.

HTTP 5xx errors have been resolved.

A fix has been implemented and admin are monitoring logs and metrics.

Mastodon Canada has been upgraded to v4.0.2+glitch. Thank you for your patience!

Mastodon Canada has been upgraded to v4.0.1+glitch. Thank you for your patience!