Back to overview
Degraded
Mastodon Local Toots Not Being Distributed
Jun 30 at 11:18pm MDT
Affected services
mstdn.ca
Resolved
Jul 01 at 03:37am MDT
We are back to normal Sidekiq job levels.
Post-mortem is that a stale Sidekiq process wasn't terminated by systemd as requested at the time of the update. The stale process was looking for a database column that no longer exists. Killing this process brought our instance back online.
Thank you for your patience!
Affected services
mstdn.ca
Created
Jun 30 at 11:18pm MDT
Please be advised we are working an error where a stuck Sidekiq process caused the instance to essentially freeze up, where local toots were not being distributed locally.
We believe a fix is in place and the instance is starting to recover. This may take some time over the next few hours.
Affected services
mstdn.ca