Back to overview
Degraded

Mastodon Local Toots Not Being Distributed

Jun 30 at 11:18pm MDT
Affected services
mstdn.ca

Resolved
Jul 01 at 03:37am MDT

We are back to normal Sidekiq job levels.

Post-mortem is that a stale Sidekiq process wasn't terminated by systemd as requested at the time of the update. The stale process was looking for a database column that no longer exists. Killing this process brought our instance back online.

Thank you for your patience!

Created
Jun 30 at 11:18pm MDT

Please be advised we are working an error where a stuck Sidekiq process caused the instance to essentially freeze up, where local toots were not being distributed locally.

We believe a fix is in place and the instance is starting to recover. This may take some time over the next few hours.