Incidents reported on status page for Mastodon Canada https://status.mstdn.ca/ https://d1lppblt9t2x15.cloudfront.net/logos/a3a7d2e9dd28909ab672f575e134baa1.png https://status.mstdn.ca/ en https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 09:00:40 -0000 https://status.mstdn.ca/incident/834332#1ec1a158e5b23c759e30e79b19c24ad57d9a3a2277ad1527906a4fcbddfd122c Maintenance completed https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 09:00:40 -0000 https://status.mstdn.ca/incident/834332#1ec1a158e5b23c759e30e79b19c24ad57d9a3a2277ad1527906a4fcbddfd122c Maintenance completed https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 05:30:40 -0000 https://status.mstdn.ca/incident/834332#79e8540536bbf85517be74a0b886fbad39e7e90d92a97a8a103b517df6b2a9d7 We will be upgrading to Mastodon 4.5.7 during this upgrade window. This would normally be a shorter window, but we are attempting to use an in-house automated script first. If it is successful, our upgrade window will be shorter than usual. If it's not successful, it may take more time. Updates will be provided here. https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 05:30:40 -0000 https://status.mstdn.ca/incident/834332#79e8540536bbf85517be74a0b886fbad39e7e90d92a97a8a103b517df6b2a9d7 We will be upgrading to Mastodon 4.5.7 during this upgrade window. This would normally be a shorter window, but we are attempting to use an in-house automated script first. If it is successful, our upgrade window will be shorter than usual. If it's not successful, it may take more time. Updates will be provided here. https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 05:30:40 -0000 https://status.mstdn.ca/incident/834332#79e8540536bbf85517be74a0b886fbad39e7e90d92a97a8a103b517df6b2a9d7 We will be upgrading to Mastodon 4.5.7 during this upgrade window. This would normally be a shorter window, but we are attempting to use an in-house automated script first. If it is successful, our upgrade window will be shorter than usual. If it's not successful, it may take more time. Updates will be provided here. https://status.mstdn.ca/incident/834332 Fri, 27 Feb 2026 05:30:40 -0000 https://status.mstdn.ca/incident/834332#79e8540536bbf85517be74a0b886fbad39e7e90d92a97a8a103b517df6b2a9d7 We will be upgrading to Mastodon 4.5.7 during this upgrade window. This would normally be a shorter window, but we are attempting to use an in-house automated script first. If it is successful, our upgrade window will be shorter than usual. If it's not successful, it may take more time. Updates will be provided here. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 11:17:29 -0000 https://status.mstdn.ca/incident/831038#9f0d144c15e54218725943d454aaa340d1284175235fc28cd388fe6793d7d4e6 Maintenance completed https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 11:17:29 -0000 https://status.mstdn.ca/incident/831038#9f0d144c15e54218725943d454aaa340d1284175235fc28cd388fe6793d7d4e6 Maintenance completed https://status.mstdn.ca/ Tue, 24 Feb 2026 09:33:26 +0000 https://status.mstdn.ca/#0f74b231eb3aeae79aea894f0d66c6106151850bd9166bff99838c710c9a63e3 mstdn.ca recovered https://status.mstdn.ca/ Tue, 24 Feb 2026 09:32:20 +0000 https://status.mstdn.ca/#7a65837cc9abe25093b76300ce4cde2d23c45c417fc8c0489a166fac9fd6fa52 Streaming API recovered https://status.mstdn.ca/ Tue, 24 Feb 2026 06:18:27 +0000 https://status.mstdn.ca/#7a65837cc9abe25093b76300ce4cde2d23c45c417fc8c0489a166fac9fd6fa52 Streaming API went down https://status.mstdn.ca/ Tue, 24 Feb 2026 06:17:21 +0000 https://status.mstdn.ca/#0f74b231eb3aeae79aea894f0d66c6106151850bd9166bff99838c710c9a63e3 mstdn.ca went down https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/incident/831038 Tue, 24 Feb 2026 05:00:00 -0000 https://status.mstdn.ca/incident/831038#997a200dabfe0d5e83b24b903d5e5c49169a8e5cb54e209b3c22e94dbfd9d41f Mastodon Canada will be taking a full and complete downtime to migrate our 450 GB database from the cloud to our servers. We're reserving four hours for this work; we don't expect to use the entire time. Updates will be provided here as they become available. https://status.mstdn.ca/ Sun, 22 Feb 2026 14:33:29 +0000 https://status.mstdn.ca/#7843358e5d442eb4adbd2ec9cd5e28a5bf456bba372a525d88bcd27150465641 mstdn.ca recovered https://status.mstdn.ca/ Sun, 22 Feb 2026 14:32:35 +0000 https://status.mstdn.ca/#a0b648c0b23f0c51f54f4165db62f95877738fa04171bfe7ed9a7e0f68248547 Streaming API recovered https://status.mstdn.ca/ Sun, 22 Feb 2026 13:59:14 +0000 https://status.mstdn.ca/#a0b648c0b23f0c51f54f4165db62f95877738fa04171bfe7ed9a7e0f68248547 Streaming API went down https://status.mstdn.ca/ Sun, 22 Feb 2026 13:58:27 +0000 https://status.mstdn.ca/#7843358e5d442eb4adbd2ec9cd5e28a5bf456bba372a525d88bcd27150465641 mstdn.ca went down https://status.mstdn.ca/ Sun, 22 Feb 2026 00:15:19 +0000 https://status.mstdn.ca/#c54c88b61defa89e6084f60a1f5c1288bdfb819ce2b5e5de720b664cdb0778bd Media CDN recovered https://status.mstdn.ca/ Sun, 22 Feb 2026 00:04:22 +0000 https://status.mstdn.ca/#c54c88b61defa89e6084f60a1f5c1288bdfb819ce2b5e5de720b664cdb0778bd Media CDN went down https://status.mstdn.ca/ Mon, 16 Feb 2026 11:58:18 +0000 https://status.mstdn.ca/#b3d379f9527d5f795004af19d72e85a5293b38f24b5d8a49c578938eb5fb8e04 Streaming API recovered https://status.mstdn.ca/ Mon, 16 Feb 2026 11:56:23 +0000 https://status.mstdn.ca/#c353d94df230ac851a075f4cd3ef33b132fba8f1603d10cdc148d6ae78f0f95b mstdn.ca recovered https://status.mstdn.ca/ Mon, 16 Feb 2026 11:53:01 +0000 https://status.mstdn.ca/#b3d379f9527d5f795004af19d72e85a5293b38f24b5d8a49c578938eb5fb8e04 Streaming API went down https://status.mstdn.ca/ Mon, 16 Feb 2026 11:51:17 +0000 https://status.mstdn.ca/#c353d94df230ac851a075f4cd3ef33b132fba8f1603d10cdc148d6ae78f0f95b mstdn.ca went down https://status.mstdn.ca/ Mon, 09 Feb 2026 23:40:24 +0000 https://status.mstdn.ca/#e56946ac2041f12f2137227a74c5e1eb8728005afd11feef9ee872cb85194f2f Streaming API recovered https://status.mstdn.ca/ Mon, 09 Feb 2026 22:16:29 +0000 https://status.mstdn.ca/#e56946ac2041f12f2137227a74c5e1eb8728005afd11feef9ee872cb85194f2f Streaming API went down https://status.mstdn.ca/ Tue, 03 Feb 2026 09:01:25 +0000 https://status.mstdn.ca/#7ce92abfc37bf61237e6f66b1e60683f2bae71ab19f387a662d29434a597f17b mstdn.ca recovered https://status.mstdn.ca/incident/814555 Tue, 03 Feb 2026 09:00:00 -0000 https://status.mstdn.ca/incident/814555#4724c74cc4b2cc1b2890e0b7d7f6e01f4f1c48b4b1d6c36995ccd84ee4696dfb Maintenance completed https://status.mstdn.ca/incident/814555 Tue, 03 Feb 2026 09:00:00 -0000 https://status.mstdn.ca/incident/814555#4724c74cc4b2cc1b2890e0b7d7f6e01f4f1c48b4b1d6c36995ccd84ee4696dfb Maintenance completed https://status.mstdn.ca/ Tue, 03 Feb 2026 08:51:21 +0000 https://status.mstdn.ca/#7ce92abfc37bf61237e6f66b1e60683f2bae71ab19f387a662d29434a597f17b mstdn.ca went down https://status.mstdn.ca/incident/814555 Tue, 03 Feb 2026 06:00:00 -0000 https://status.mstdn.ca/incident/814555#be9b8f8dc6efacab43a8c56671105caa2055219b941249a0387e6a4468aa9ed9 We will be taking a brief downtime at the specified time to upgrade to Mastodon v4.5.5. This may result in a brief outage or degraded performance during the upgrade window. https://status.mstdn.ca/incident/814555 Tue, 03 Feb 2026 06:00:00 -0000 https://status.mstdn.ca/incident/814555#be9b8f8dc6efacab43a8c56671105caa2055219b941249a0387e6a4468aa9ed9 We will be taking a brief downtime at the specified time to upgrade to Mastodon v4.5.5. This may result in a brief outage or degraded performance during the upgrade window. https://status.mstdn.ca/incident/806156 Fri, 23 Jan 2026 04:19:00 -0000 https://status.mstdn.ca/incident/806156#8043b5c2448f1649b54635b60427f93dd4bfeb17abf25867a0b2c4cbe4ada9bd After a rather eventless day, we are declaring this event resolved. https://status.mstdn.ca/incident/806156 Fri, 23 Jan 2026 04:19:00 -0000 https://status.mstdn.ca/incident/806156#8043b5c2448f1649b54635b60427f93dd4bfeb17abf25867a0b2c4cbe4ada9bd After a rather eventless day, we are declaring this event resolved. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 10:42:00 -0000 https://status.mstdn.ca/incident/806156#6d02a97adb93040a14b9153d34ec8ca4a1f032f5e2d899777a13c4db4be6b269 We were very successful tonight in getting the database under control. It took a gargantuan effort, and we thank everyone in the Discord for all their help. This was a cross-instance effort, and we're so humbled by that. @RichardNairn, @andrewdemarsh, @quaff, @ThaMunsta, @controlc, among so many others put our brains together. As it turns out, we were missing a couple of indexes in the (now 364GB, was 480GB) database, autovacuum wasn't cleaning up a few tables frequently enough, and our concurrency (connections) with the database were too high. All of this grew to a head last Friday, like if there was too much water for a waterfall. There are about 600,000 jobs left for the instance to process to catch up with the fediverse, so if you're notifications or posts are two hours behind, that's why. That latency is decreasing with time. We are hopeful that come the morning when our traffic comes back, we're still in great shape. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 10:42:00 -0000 https://status.mstdn.ca/incident/806156#6d02a97adb93040a14b9153d34ec8ca4a1f032f5e2d899777a13c4db4be6b269 We were very successful tonight in getting the database under control. It took a gargantuan effort, and we thank everyone in the Discord for all their help. This was a cross-instance effort, and we're so humbled by that. @RichardNairn, @andrewdemarsh, @quaff, @ThaMunsta, @controlc, among so many others put our brains together. As it turns out, we were missing a couple of indexes in the (now 364GB, was 480GB) database, autovacuum wasn't cleaning up a few tables frequently enough, and our concurrency (connections) with the database were too high. All of this grew to a head last Friday, like if there was too much water for a waterfall. There are about 600,000 jobs left for the instance to process to catch up with the fediverse, so if you're notifications or posts are two hours behind, that's why. That latency is decreasing with time. We are hopeful that come the morning when our traffic comes back, we're still in great shape. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 06:13:00 -0000 https://status.mstdn.ca/incident/806156#58bbc10e04459e09432e6cb5158fd380f8f9d189ca3ceb6204fcba77ca217c67 We've made great progress this evening - with web response times returning to normal. Third party apps are also working again. At this time, the instance has to catch up with what was missed while the database was locked up. It's estimated that we're about five hours behind. This catch-up work will work through the night. With hope, a final resolution update will be provided tomorrow. Thank you, again, for your patience as we worked through this complex problem. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 06:13:00 -0000 https://status.mstdn.ca/incident/806156#58bbc10e04459e09432e6cb5158fd380f8f9d189ca3ceb6204fcba77ca217c67 We've made great progress this evening - with web response times returning to normal. Third party apps are also working again. At this time, the instance has to catch up with what was missed while the database was locked up. It's estimated that we're about five hours behind. This catch-up work will work through the night. With hope, a final resolution update will be provided tomorrow. Thank you, again, for your patience as we worked through this complex problem. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 04:59:00 -0000 https://status.mstdn.ca/incident/806156#b3d6f06229343d0d09f7a77ebd38c5c350f7788bf52fbd3cab2e97a1dcc0fd61 We are still working through this issue. Earlier today, we discovered a stuck trigger in the database. Once removed, the database started accepting data again. However, there's a lot of jobs to catch up on, which the instance is doing now. We're continuing to look at ways to optimize the database. https://status.mstdn.ca/incident/806156 Thu, 22 Jan 2026 04:59:00 -0000 https://status.mstdn.ca/incident/806156#b3d6f06229343d0d09f7a77ebd38c5c350f7788bf52fbd3cab2e97a1dcc0fd61 We are still working through this issue. Earlier today, we discovered a stuck trigger in the database. Once removed, the database started accepting data again. However, there's a lot of jobs to catch up on, which the instance is doing now. We're continuing to look at ways to optimize the database. https://status.mstdn.ca/incident/806156 Wed, 21 Jan 2026 09:33:00 -0000 https://status.mstdn.ca/incident/806156#7ef4cb93747e8aa3efeca5ca06e203a2d19cbcfbe4c5ad390b2431eb4335cccb Thanks to all that joined the "all-hands" call this evening. We looked at logs, made some changes, and though we ran into a storage limit, the database is now recovering from that. It's hopeful that when the database has recovered, we will see an improvement in response times. https://status.mstdn.ca/incident/806156 Wed, 21 Jan 2026 09:33:00 -0000 https://status.mstdn.ca/incident/806156#7ef4cb93747e8aa3efeca5ca06e203a2d19cbcfbe4c5ad390b2431eb4335cccb Thanks to all that joined the "all-hands" call this evening. We looked at logs, made some changes, and though we ran into a storage limit, the database is now recovering from that. It's hopeful that when the database has recovered, we will see an improvement in response times. https://status.mstdn.ca/incident/806156 Tue, 20 Jan 2026 08:57:00 -0000 https://status.mstdn.ca/incident/806156#d7ed56689703691e2c19d4573e6907808dd9aed7a536ec4b3282f1f547ce030b We attempted to cut over to the new database server and it didn't go as planned. We're looking at logs and will attempt again. https://status.mstdn.ca/incident/806156 Tue, 20 Jan 2026 08:57:00 -0000 https://status.mstdn.ca/incident/806156#d7ed56689703691e2c19d4573e6907808dd9aed7a536ec4b3282f1f547ce030b We attempted to cut over to the new database server and it didn't go as planned. We're looking at logs and will attempt again. https://status.mstdn.ca/ Tue, 20 Jan 2026 08:42:27 +0000 https://status.mstdn.ca/#25af348cb810d1b0fb41e83e43b622cfdc7238228fa172dd65785c69175107fc Streaming API recovered https://status.mstdn.ca/ Tue, 20 Jan 2026 08:36:26 +0000 https://status.mstdn.ca/#25af348cb810d1b0fb41e83e43b622cfdc7238228fa172dd65785c69175107fc Streaming API went down https://status.mstdn.ca/incident/806156 Tue, 20 Jan 2026 07:16:00 -0000 https://status.mstdn.ca/incident/806156#622cc61a190a1753d416c21721aaa66e78ff027fca3169b9bf31689b55ec77ba The new database server is nearly ready - all data has been copied, however it's still sorting out and organizing all the data. (PostgreSQL VACUUM) We expect to test the switch over sometime overnight. Thank you all for your patience. https://status.mstdn.ca/incident/806156 Tue, 20 Jan 2026 07:16:00 -0000 https://status.mstdn.ca/incident/806156#622cc61a190a1753d416c21721aaa66e78ff027fca3169b9bf31689b55ec77ba The new database server is nearly ready - all data has been copied, however it's still sorting out and organizing all the data. (PostgreSQL VACUUM) We expect to test the switch over sometime overnight. Thank you all for your patience. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 14:14:00 -0000 https://status.mstdn.ca/incident/806156#9bb8e3b93eeaa7eb70cc5bf93f110e78c93b131323aad18f78e82b3ad0f6c4c2 All indications show that we have fully ingested the old server into the new one, however we want to give the new server some time to stabilize. We will attempt a cutover this evening at an undeclared time as to prevent a rush of traffic. The outcome of this operation will be declared here. Note that this is a major operation and it's not being taken lightly. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 14:14:00 -0000 https://status.mstdn.ca/incident/806156#9bb8e3b93eeaa7eb70cc5bf93f110e78c93b131323aad18f78e82b3ad0f6c4c2 All indications show that we have fully ingested the old server into the new one, however we want to give the new server some time to stabilize. We will attempt a cutover this evening at an undeclared time as to prevent a rush of traffic. The outcome of this operation will be declared here. Note that this is a major operation and it's not being taken lightly. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 13:59:00 -0000 https://status.mstdn.ca/incident/806156#900973e1c4e340accbed5f7d3a959d82cf1f32a4426c0e48f9e62339eb1712bb We're getting very close to resolving this - the new database server has received nearly all of the data from the old server. We're giving it some breathing time to make sure that it has organized all this data and is ready for prime time. If everything goes to plan, we will take a brief downtime this evening to switch over to the new database. Thank you again for your patience. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 13:59:00 -0000 https://status.mstdn.ca/incident/806156#900973e1c4e340accbed5f7d3a959d82cf1f32a4426c0e48f9e62339eb1712bb We're getting very close to resolving this - the new database server has received nearly all of the data from the old server. We're giving it some breathing time to make sure that it has organized all this data and is ready for prime time. If everything goes to plan, we will take a brief downtime this evening to switch over to the new database. Thank you again for your patience. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 09:43:00 -0000 https://status.mstdn.ca/incident/806156#eb1731b580bdf638affde42bbb0eeff8a463003762e2e6b5b70d104b6f3e8d8f We're getting really close to what we hope is resolution. We've started a live stream to show where the transfer is - until the top right status screen shows less than 5Mb of transfer, we are still in replication. Once we're flatlined, we will be ready to switch over to the new server. https://www.youtube.com/live/SMpPe3wy1-k Thank you again for your patience. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 09:43:00 -0000 https://status.mstdn.ca/incident/806156#eb1731b580bdf638affde42bbb0eeff8a463003762e2e6b5b70d104b6f3e8d8f We're getting really close to what we hope is resolution. We've started a live stream to show where the transfer is - until the top right status screen shows less than 5Mb of transfer, we are still in replication. Once we're flatlined, we will be ready to switch over to the new server. https://www.youtube.com/live/SMpPe3wy1-k Thank you again for your patience. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 08:41:00 -0000 https://status.mstdn.ca/incident/806156#809a7ecd8ce7ac67541c442777c40adc9dda3082264f84430a51233f65ea7e0b After the initial database import, the new database server is now working through making sure everything matches between the old server and new. Once this is complete, we'll switch over and the hope is, our latency issues will be resolved and we'll be snappier than ever. The replication has been going for about three hours, and being it's transferred 36GB, I feel we're about three hours away from it being done. So - with hope - we should be able to take a brief downtime, change configuration files, and switch over to the new database tomorrow afternoon. Thanks all for your patience. I know this weekend has been difficult. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 08:41:00 -0000 https://status.mstdn.ca/incident/806156#809a7ecd8ce7ac67541c442777c40adc9dda3082264f84430a51233f65ea7e0b After the initial database import, the new database server is now working through making sure everything matches between the old server and new. Once this is complete, we'll switch over and the hope is, our latency issues will be resolved and we'll be snappier than ever. The replication has been going for about three hours, and being it's transferred 36GB, I feel we're about three hours away from it being done. So - with hope - we should be able to take a brief downtime, change configuration files, and switch over to the new database tomorrow afternoon. Thanks all for your patience. I know this weekend has been difficult. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 05:47:00 -0000 https://status.mstdn.ca/incident/806156#2640f4dbb77b44b616dce0c9988cbe40ee0f3dc42d91603a1d8de9f1389449c5 Initial ingest into the new database server is complete. We're now waiting for the new server to catch up with anything that needs to be replicated since our point-in-time data dump. https://status.mstdn.ca/incident/806156 Mon, 19 Jan 2026 05:47:00 -0000 https://status.mstdn.ca/incident/806156#2640f4dbb77b44b616dce0c9988cbe40ee0f3dc42d91603a1d8de9f1389449c5 Initial ingest into the new database server is complete. We're now waiting for the new server to catch up with anything that needs to be replicated since our point-in-time data dump. https://status.mstdn.ca/incident/806156 Sun, 18 Jan 2026 06:08:00 -0000 https://status.mstdn.ca/incident/806156#47cc93c241a06a7e7ed93bd887b84089c361b77206e1a46a76ec056c228a6732 The initial ingest into the new database server is still underway. We're monitoring its process closely and will share more information as it becomes available. https://status.mstdn.ca/incident/806156 Sun, 18 Jan 2026 06:08:00 -0000 https://status.mstdn.ca/incident/806156#47cc93c241a06a7e7ed93bd887b84089c361b77206e1a46a76ec056c228a6732 The initial ingest into the new database server is still underway. We're monitoring its process closely and will share more information as it becomes available. https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 20:24:00 -0000 https://status.mstdn.ca/incident/806156#b5bd268d19116d22cb6db909bc5b345b1c347ed949b090ffde1d4436369416d4 The initial database transfer is still being ingested by the new server. We recognize that this is challenging for our community and thank you for your patience. https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 20:24:00 -0000 https://status.mstdn.ca/incident/806156#b5bd268d19116d22cb6db909bc5b345b1c347ed949b090ffde1d4436369416d4 The initial database transfer is still being ingested by the new server. We recognize that this is challenging for our community and thank you for your patience. https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 10:45:00 -0000 https://status.mstdn.ca/incident/806156#55e5ea09cce704872830adba4c2362b725ecde75f8250948c8e3de61d5f4f011 We have set the gears in motion to move the PostgreSQL database away from DigitalOcean, where the latency is. This database is ~64GB, and is currently being ingested to a new database local to the rest of our resources. This process may take some time, and we appreciate your patience. https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 10:45:00 -0000 https://status.mstdn.ca/incident/806156#55e5ea09cce704872830adba4c2362b725ecde75f8250948c8e3de61d5f4f011 We have set the gears in motion to move the PostgreSQL database away from DigitalOcean, where the latency is. This database is ~64GB, and is currently being ingested to a new database local to the rest of our resources. This process may take some time, and we appreciate your patience. https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 09:57:00 -0000 https://status.mstdn.ca/incident/806156#b34c2c173e28bb62da4f3c8f3ce9e7ec9a8e2c4758ffd39bbdb02d146ec43a76 We've noted that latency has come down overnight, however we are still working on putting in a permanent fix. This requires processing over 66GB of data. We will continue to update as more information arises. Thank you for your patience! https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 09:57:00 -0000 https://status.mstdn.ca/incident/806156#b34c2c173e28bb62da4f3c8f3ce9e7ec9a8e2c4758ffd39bbdb02d146ec43a76 We've noted that latency has come down overnight, however we are still working on putting in a permanent fix. This requires processing over 66GB of data. We will continue to update as more information arises. Thank you for your patience! https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 06:34:00 -0000 https://status.mstdn.ca/incident/806156#e8532a362a3d3bd34c77460ba00c4e0c17c6cab0bca29c0545bd6af52a4cdfca We are aware of degrated performance when accessing the instance. We believe to have found the cause and are working to resolve it. Thank you for your patience! https://status.mstdn.ca/incident/806156 Sat, 17 Jan 2026 06:34:00 -0000 https://status.mstdn.ca/incident/806156#e8532a362a3d3bd34c77460ba00c4e0c17c6cab0bca29c0545bd6af52a4cdfca We are aware of degrated performance when accessing the instance. We believe to have found the cause and are working to resolve it. Thank you for your patience! https://status.mstdn.ca/ Thu, 08 Jan 2026 16:22:22 +0000 https://status.mstdn.ca/#7eb87cea69ca41f3957fbf4263a01f0a429216785086340a4f49fa37c3c333d3 Streaming API recovered https://status.mstdn.ca/ Thu, 08 Jan 2026 16:21:21 +0000 https://status.mstdn.ca/#a08e5ba403cda7ff9e1cdc420aa61a3a676f504d5b318f65947a37fb6b64a336 mstdn.ca recovered https://status.mstdn.ca/ Thu, 08 Jan 2026 16:14:21 +0000 https://status.mstdn.ca/#7eb87cea69ca41f3957fbf4263a01f0a429216785086340a4f49fa37c3c333d3 Streaming API went down https://status.mstdn.ca/ Thu, 08 Jan 2026 16:13:17 +0000 https://status.mstdn.ca/#a08e5ba403cda7ff9e1cdc420aa61a3a676f504d5b318f65947a37fb6b64a336 mstdn.ca went down https://status.mstdn.ca/ Thu, 11 Dec 2025 12:03:51 +0000 https://status.mstdn.ca/#90aaf7fc37ee4c7b5a27422a5ffa53aa01babbe227178be65d10a62dcb0d9511 Streaming API recovered https://status.mstdn.ca/ Thu, 11 Dec 2025 10:13:51 +0000 https://status.mstdn.ca/#90aaf7fc37ee4c7b5a27422a5ffa53aa01babbe227178be65d10a62dcb0d9511 Streaming API went down https://status.mstdn.ca/incident/730885 Thu, 25 Sep 2025 06:24:29 -0000 https://status.mstdn.ca/incident/730885#36b3c7ed0244b3680f69eb89cfc759e9907d288c5699cf895dc680ccd297aece Maintenance completed https://status.mstdn.ca/incident/730885 Thu, 25 Sep 2025 04:00:13 -0000 https://status.mstdn.ca/incident/730885#f40af9bb211cc675617343d11456dd57853dd240d3296c9cf54d3ad774f2151b Please be advised that we will be upgrading to Mastodon 4.4.5 during the window below. There may be periods of instability during this time. https://status.mstdn.ca/incident/730885 Thu, 25 Sep 2025 04:00:13 -0000 https://status.mstdn.ca/incident/730885#f40af9bb211cc675617343d11456dd57853dd240d3296c9cf54d3ad774f2151b Please be advised that we will be upgrading to Mastodon 4.4.5 during the window below. There may be periods of instability during this time. https://status.mstdn.ca/incident/730885 Thu, 25 Sep 2025 04:00:13 -0000 https://status.mstdn.ca/incident/730885#f40af9bb211cc675617343d11456dd57853dd240d3296c9cf54d3ad774f2151b Please be advised that we will be upgrading to Mastodon 4.4.5 during the window below. There may be periods of instability during this time. https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 07:27:08 -0000 https://status.mstdn.ca/incident/713240#8560177f40e9bb1b02ce186befac21101db0bf009afb80ffab53eb9abc8573d8 Maintenance completed https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 07:27:08 -0000 https://status.mstdn.ca/incident/713240#8560177f40e9bb1b02ce186befac21101db0bf009afb80ffab53eb9abc8573d8 Maintenance completed https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 04:30:50 -0000 https://status.mstdn.ca/incident/713240#88486a111fe8096ee19ef020ecb61a2e8991e7915c6d91fb5857e26efc188481 Please be advised that Mastodon Canada will take a downtime sometime between the specified times to perform required network maintenance. https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 04:30:50 -0000 https://status.mstdn.ca/incident/713240#88486a111fe8096ee19ef020ecb61a2e8991e7915c6d91fb5857e26efc188481 Please be advised that Mastodon Canada will take a downtime sometime between the specified times to perform required network maintenance. https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 04:30:50 -0000 https://status.mstdn.ca/incident/713240#88486a111fe8096ee19ef020ecb61a2e8991e7915c6d91fb5857e26efc188481 Please be advised that Mastodon Canada will take a downtime sometime between the specified times to perform required network maintenance. https://status.mstdn.ca/incident/713240 Thu, 28 Aug 2025 04:30:50 -0000 https://status.mstdn.ca/incident/713240#88486a111fe8096ee19ef020ecb61a2e8991e7915c6d91fb5857e26efc188481 Please be advised that Mastodon Canada will take a downtime sometime between the specified times to perform required network maintenance. https://status.mstdn.ca/incident/616336 Thu, 10 Jul 2025 09:12:21 -0000 https://status.mstdn.ca/incident/616336#e3874c8651ad1fb0d37be0771df646d355004ca5409d84bd5d1b4d1f625f6f72 Maintenance completed https://status.mstdn.ca/incident/616336 Thu, 10 Jul 2025 06:00:00 -0000 https://status.mstdn.ca/incident/616336#5b8fbc8435e0a53292506a1699d6b993b16f37624515101ee29b1b8dfa14c7df Please be advised Mastodon Canada will take a brief downtime at the time specified below to upgrade to version 4.4.0+glitch. A full run-down of the new features and patches is available here: https://blog.joinmastodon.org/2025/07/mastodon-4.4/ https://status.mstdn.ca/incident/616336 Thu, 10 Jul 2025 06:00:00 -0000 https://status.mstdn.ca/incident/616336#5b8fbc8435e0a53292506a1699d6b993b16f37624515101ee29b1b8dfa14c7df Please be advised Mastodon Canada will take a brief downtime at the time specified below to upgrade to version 4.4.0+glitch. A full run-down of the new features and patches is available here: https://blog.joinmastodon.org/2025/07/mastodon-4.4/ https://status.mstdn.ca/incident/616336 Thu, 10 Jul 2025 06:00:00 -0000 https://status.mstdn.ca/incident/616336#5b8fbc8435e0a53292506a1699d6b993b16f37624515101ee29b1b8dfa14c7df Please be advised Mastodon Canada will take a brief downtime at the time specified below to upgrade to version 4.4.0+glitch. A full run-down of the new features and patches is available here: https://blog.joinmastodon.org/2025/07/mastodon-4.4/ https://status.mstdn.ca/incident/612191 Tue, 01 Jul 2025 09:37:00 -0000 https://status.mstdn.ca/incident/612191#f4c42b3543891ba3b365b52754a951b4f40044759efd51836ddb1ccc511b0df2 We are back to normal Sidekiq job levels. Post-mortem is that a stale Sidekiq process wasn't terminated by systemd as requested at the time of the update. The stale process was looking for a database column that no longer exists. Killing this process brought our instance back online. Thank you for your patience! https://status.mstdn.ca/incident/612191 Tue, 01 Jul 2025 05:18:00 -0000 https://status.mstdn.ca/incident/612191#d171ef7cf2ee7c8a5ffe26f3bc85a4e3243dcff027b7d75beb56857686b80a9f Please be advised we are working an error where a stuck Sidekiq process caused the instance to essentially freeze up, where local toots were not being distributed locally. We believe a fix is in place and the instance is starting to recover. This may take some time over the next few hours. https://status.mstdn.ca/incident/609838 Mon, 30 Jun 2025 05:43:33 -0000 https://status.mstdn.ca/incident/609838#ed4c2ba0d932fe356ce85690db23bdaa0a3f1a39e2e0f67fcb88bd538297fefe Maintenance completed https://status.mstdn.ca/incident/609838 Mon, 30 Jun 2025 04:00:37 -0000 https://status.mstdn.ca/incident/609838#7b28bcedc446f300d3c6d3c9b43e0ada86d0a8bdc8f8e294ae845dee98eca60e Please be advised that Mastodon Canada will take some time for maintenance during the specified window. https://status.mstdn.ca/incident/609838 Mon, 30 Jun 2025 04:00:37 -0000 https://status.mstdn.ca/incident/609838#7b28bcedc446f300d3c6d3c9b43e0ada86d0a8bdc8f8e294ae845dee98eca60e Please be advised that Mastodon Canada will take some time for maintenance during the specified window. https://status.mstdn.ca/incident/544522 Tue, 15 Apr 2025 06:00:00 -0000 https://status.mstdn.ca/incident/544522#be4e417895a60fd42970902cf2e4dfe2690ef4544a544ccab36704646efe40fe Maintenance completed https://status.mstdn.ca/incident/544522 Tue, 15 Apr 2025 05:30:00 -0000 https://status.mstdn.ca/incident/544522#257b016ef2656de15dfb64111e681faf27cd34e41c22c560d6b507c2d7c67c75 Mastodon Canada will attempt to move the PostgreSQL database from DigitalOcean locally during the time allotted. Members are asked not to use the instance during this time. https://status.mstdn.ca/incident/544522 Tue, 15 Apr 2025 05:30:00 -0000 https://status.mstdn.ca/incident/544522#257b016ef2656de15dfb64111e681faf27cd34e41c22c560d6b507c2d7c67c75 Mastodon Canada will attempt to move the PostgreSQL database from DigitalOcean locally during the time allotted. Members are asked not to use the instance during this time. https://status.mstdn.ca/incident/544522 Tue, 15 Apr 2025 05:30:00 -0000 https://status.mstdn.ca/incident/544522#257b016ef2656de15dfb64111e681faf27cd34e41c22c560d6b507c2d7c67c75 Mastodon Canada will attempt to move the PostgreSQL database from DigitalOcean locally during the time allotted. Members are asked not to use the instance during this time. https://status.mstdn.ca/incident/542478 Wed, 09 Apr 2025 11:30:17 -0000 https://status.mstdn.ca/incident/542478#22506af6974d234542e10c38113041493e8ce55035e718306d4b91deddcaa978 Maintenance completed https://status.mstdn.ca/incident/542478 Wed, 09 Apr 2025 10:00:17 -0000 https://status.mstdn.ca/incident/542478#bbc8b79709377bfa1f18570444a429aa94c98ac204e04827c2717b381af8d06c Mastodon Canada is performing an upgrade during the time specified. A downtime is expected during this time. https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 12:40:05 -0000 https://status.mstdn.ca/incident/499043#8b92618be2bbd09f850ebe3e3054a4fef42ff955181cde41675d6d1922b26ded Maintenance completed https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 12:40:05 -0000 https://status.mstdn.ca/incident/499043#8b92618be2bbd09f850ebe3e3054a4fef42ff955181cde41675d6d1922b26ded Maintenance completed https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 08:01:16 -0000 https://status.mstdn.ca/incident/499043#39eee202e14e9be46d25616fda6a9cf623c4ce79856d80f6ed178f29c397c2c9 Please note that our ISP, TELUS, has advised Mastodon Canada of a potential overnight outage, which will be "completed by early morning." In the event of an outage, we will automatically fail-over to our secondary ISP, however the service may be degraded due to bandwidth constraints on this secondary ISP. For more information about the outage, please visit TELUS' Service Status Site: https://www.telus.com/en/ab/outages https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 08:01:16 -0000 https://status.mstdn.ca/incident/499043#39eee202e14e9be46d25616fda6a9cf623c4ce79856d80f6ed178f29c397c2c9 Please note that our ISP, TELUS, has advised Mastodon Canada of a potential overnight outage, which will be "completed by early morning." In the event of an outage, we will automatically fail-over to our secondary ISP, however the service may be degraded due to bandwidth constraints on this secondary ISP. For more information about the outage, please visit TELUS' Service Status Site: https://www.telus.com/en/ab/outages https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 08:01:16 -0000 https://status.mstdn.ca/incident/499043#39eee202e14e9be46d25616fda6a9cf623c4ce79856d80f6ed178f29c397c2c9 Please note that our ISP, TELUS, has advised Mastodon Canada of a potential overnight outage, which will be "completed by early morning." In the event of an outage, we will automatically fail-over to our secondary ISP, however the service may be degraded due to bandwidth constraints on this secondary ISP. For more information about the outage, please visit TELUS' Service Status Site: https://www.telus.com/en/ab/outages https://status.mstdn.ca/incident/499043 Tue, 21 Jan 2025 08:01:16 -0000 https://status.mstdn.ca/incident/499043#39eee202e14e9be46d25616fda6a9cf623c4ce79856d80f6ed178f29c397c2c9 Please note that our ISP, TELUS, has advised Mastodon Canada of a potential overnight outage, which will be "completed by early morning." In the event of an outage, we will automatically fail-over to our secondary ISP, however the service may be degraded due to bandwidth constraints on this secondary ISP. For more information about the outage, please visit TELUS' Service Status Site: https://www.telus.com/en/ab/outages https://status.mstdn.ca/incident/496867 Sun, 19 Jan 2025 10:00:13 -0000 https://status.mstdn.ca/incident/496867#459d4df2c108fa9fa14dcfaf2db04ff095e51eeec04ca5d91b66e1731fc27042 Maintenance completed https://status.mstdn.ca/incident/496867 Sun, 19 Jan 2025 10:00:13 -0000 https://status.mstdn.ca/incident/496867#459d4df2c108fa9fa14dcfaf2db04ff095e51eeec04ca5d91b66e1731fc27042 Maintenance completed https://status.mstdn.ca/incident/496867 Sun, 19 Jan 2025 05:00:13 -0000 https://status.mstdn.ca/incident/496867#ed84893180c831c63d27336d64943f2e832167afa3e3b6bf27f741491661dc3b Please be advised we will be performing system upgrades including Unifi OS, Unifi Network, Proxmox, OS updates, and upgrading Mastodon to v4.3.3+glitch. There is a potential for instability during this time. https://status.mstdn.ca/incident/496867 Sun, 19 Jan 2025 05:00:13 -0000 https://status.mstdn.ca/incident/496867#ed84893180c831c63d27336d64943f2e832167afa3e3b6bf27f741491661dc3b Please be advised we will be performing system upgrades including Unifi OS, Unifi Network, Proxmox, OS updates, and upgrading Mastodon to v4.3.3+glitch. There is a potential for instability during this time. https://status.mstdn.ca/incident/455615 Tue, 05 Nov 2024 07:38:36 -0000 https://status.mstdn.ca/incident/455615#5f2e4d62767c2dd42f418aa3a48f167f8aa4cace24d9873e9adcc5a0a7317a79 Maintenance completed https://status.mstdn.ca/incident/455615 Tue, 05 Nov 2024 06:30:46 -0000 https://status.mstdn.ca/incident/455615#b0a0df63d3cf78b36c2c56f814ddd7e3f0d15ecc670894f7da7fbc94f06305d2 Mastodon Canada will undergo system updates, addressing certain underlying security updates and a bug-fix Mastodon update during this window. https://status.mstdn.ca/incident/455615 Tue, 05 Nov 2024 06:30:46 -0000 https://status.mstdn.ca/incident/455615#b0a0df63d3cf78b36c2c56f814ddd7e3f0d15ecc670894f7da7fbc94f06305d2 Mastodon Canada will undergo system updates, addressing certain underlying security updates and a bug-fix Mastodon update during this window. https://status.mstdn.ca/incident/443380 Thu, 17 Oct 2024 05:09:00 -0000 https://status.mstdn.ca/incident/443380#8af88857c57391053998ce3aefdd9482241ca6e0778dca81b9d10687244cd22e We've resolved the Streaming API problem! It's now back up and running. https://status.mstdn.ca/incident/443380 Wed, 16 Oct 2024 06:36:00 -0000 https://status.mstdn.ca/incident/443380#ce81962b2ff009055cb54a42055c5a4470df18f88a6c04cedaee09f11c50b8ec We had about an hour latency on the "default" queue today. Resources have been tweaked and we're caught up again. The Streaming API remains offline but we're making progress. Thanks again for your patience. https://status.mstdn.ca/incident/443380 Tue, 15 Oct 2024 09:08:00 -0000 https://status.mstdn.ca/incident/443380#7d85620c09c8e6e8aa917dfe734fee277e9c43b008053b6aa5ef599a2b4f5ec8 The instance has been optimized and queues have returned to normal. The Streaming API continues to be a challenge but are working on resolving. Until then, the service is in "slow mode" but shouldn't impact day-to-day use. https://status.mstdn.ca/incident/443380 Sun, 13 Oct 2024 06:18:00 -0000 https://status.mstdn.ca/incident/443380#2160876d1cca8e02a9c7f4b8c8d2c1b3321f981edfcf435d1c9444b53bbb67d6 The instance has caught up with the job back-up and is no longer latent. The Streaming API issue remains. Thank you for your continued patience. https://status.mstdn.ca/incident/443380 Sat, 12 Oct 2024 21:18:00 -0000 https://status.mstdn.ca/incident/443380#b818ac41e99bf81c73816ea30af497f59ac2be9becaae51e1ebd2ed105ca5e12 We identified a problem with the Puma web configuration and have put in a fix, this has sped up the service significantly. We're watching for any continued errors. The Streaming API continues to be down. Thank you for your patience! https://status.mstdn.ca/incident/443380 Sat, 12 Oct 2024 08:25:00 -0000 https://status.mstdn.ca/incident/443380#47279233cd3680f84f232e5bc51e7be318c92f972c66db62d43df3e54abe41f2 Post the upgrade to v4.3+glitch, we're experiencing problems with the Streaming API and notice the web app is not responding as quickly as we'd like. We are investigating and hope to have a fix in place soon. In the mean time, due to live updating isn't available, please (sparingly) refresh your browser to retrieve new toots. https://status.mstdn.ca/incident/442863 Sat, 12 Oct 2024 08:00:07 -0000 https://status.mstdn.ca/incident/442863#c76813fe3dcef1c5df009d1ab2ed61ba7f555d6be40aca3038be5c4ebaf7aa35 Maintenance completed https://status.mstdn.ca/incident/442863 Sat, 12 Oct 2024 04:00:00 -0000 https://status.mstdn.ca/incident/442863#849c4bae3f845fa2a92fd68d0782b10686b32131c14e5acbb43554f7c7ba19fe Mastodon Canada will take a downtime during the scheduled time to upgrade to v4.3.0+glitch. There may be intermittent issues with accessing the community during the duration of this upgrade. https://status.mstdn.ca/incident/442863 Sat, 12 Oct 2024 04:00:00 -0000 https://status.mstdn.ca/incident/442863#849c4bae3f845fa2a92fd68d0782b10686b32131c14e5acbb43554f7c7ba19fe Mastodon Canada will take a downtime during the scheduled time to upgrade to v4.3.0+glitch. There may be intermittent issues with accessing the community during the duration of this upgrade. https://status.mstdn.ca/incident/394754 Sat, 13 Jul 2024 05:59:00 -0000 https://status.mstdn.ca/incident/394754#674b947783531ec3cdc994251078179e9e94e8c26336ebfe943092222797e766 Maintenance completed https://status.mstdn.ca/incident/394754 Sun, 07 Jul 2024 09:00:00 -0000 https://status.mstdn.ca/incident/394754#389428b60bf68efa8777767f0552812330f65780ecef194c8dd685f029c58e2b We've had issues migrating to v4.3.0-alpha.5+glitch due to legacy configuration. I'm working to rebuild the entire instance stack from scratch which will set us up for success and resilience not only now but down the road. This work is going to be done over the next ~5 days, and you should expect instance stability to be up and down, but endeavouring every effort to reduce the impact. You may receive errors during this time, and I really appreciate your patience! This will also finally move the web services over to OVH from Digital Ocean, reducing $102/mo in server costs. I'll keep this thread up to date as this work progresses, particularly if we experience a downtime. https://status.mstdn.ca/incident/393756 Sat, 06 Jul 2024 09:00:00 -0000 https://status.mstdn.ca/incident/393756#a01a00475db0ada5406ab91991fffc39a50140fdf8ee6bdc17423e6a2432f390 Maintenance completed https://status.mstdn.ca/incident/393756 Sat, 06 Jul 2024 04:00:00 -0000 https://status.mstdn.ca/incident/393756#5e7f57f750ec3910fb43e1b17ca4fce4d4b7fc02d531689f34d8c1f875e8e37d Mastodon Canada will be taking a downtime to upgrade the instance and mitigate a critical security issue identified by the Mastodon developers. If time allows, we will also upgrade our Elk service. UPDATE: This maintenance has been POSTPONED until Friday evening. https://status.mstdn.ca/incident/325737 Tue, 13 Feb 2024 01:45:00 -0000 https://status.mstdn.ca/incident/325737#e44bd6d2b21f3064ad5e7acc716b2354550789a9304102bb85b98fa24464207f Root cause identified and fixed. Thank you for your patience! https://status.mstdn.ca/incident/325737 Mon, 12 Feb 2024 16:23:00 -0000 https://status.mstdn.ca/incident/325737#56d4791b0e785d679bf896dc89446d3d8546f5ed0bd73948399042bbc3e2766d We may have identified a cause and have implemented a fix. We are monitoring the situation. https://status.mstdn.ca/incident/325737 Mon, 12 Feb 2024 15:15:00 -0000 https://status.mstdn.ca/incident/325737#728346e61ef8d5011e9c6b03571218f5f72f90b2ba8474e98c73218e1f2fbdc1 The instance is facing intermittent degradation of service this morning. We are working on identifying the cause. https://status.mstdn.ca/incident/309776 Tue, 09 Jan 2024 07:48:45 -0000 https://status.mstdn.ca/incident/309776#15d1ce5ec86a53445f1f89182a763ecb81608bb37219097ad69be08ad330e39c Maintenance completed https://status.mstdn.ca/incident/309776 Tue, 09 Jan 2024 06:00:00 -0000 https://status.mstdn.ca/incident/309776#3a2f2b2b5c2b88712ccc6d2edc807afc52d3af4f1abf9ab3df5a3832dbbafadc Please be advised that the instance will be upgraded to v4.2.3+glitch during the specified maintenance window. During this time, community members may experience periods of instability or downtime as systems are upgraded. If time allows, we will also be transitioning our Sidekiq service from DigitalOcean to our dedicated OVH server. Updates will be shared on our status page as maintenance progresses. https://status.mstdn.ca/incident/309776 Tue, 09 Jan 2024 06:00:00 -0000 https://status.mstdn.ca/incident/309776#3a2f2b2b5c2b88712ccc6d2edc807afc52d3af4f1abf9ab3df5a3832dbbafadc Please be advised that the instance will be upgraded to v4.2.3+glitch during the specified maintenance window. During this time, community members may experience periods of instability or downtime as systems are upgraded. If time allows, we will also be transitioning our Sidekiq service from DigitalOcean to our dedicated OVH server. Updates will be shared on our status page as maintenance progresses. https://status.mstdn.ca/incident/309776 Tue, 09 Jan 2024 06:00:00 -0000 https://status.mstdn.ca/incident/309776#3a2f2b2b5c2b88712ccc6d2edc807afc52d3af4f1abf9ab3df5a3832dbbafadc Please be advised that the instance will be upgraded to v4.2.3+glitch during the specified maintenance window. During this time, community members may experience periods of instability or downtime as systems are upgraded. If time allows, we will also be transitioning our Sidekiq service from DigitalOcean to our dedicated OVH server. Updates will be shared on our status page as maintenance progresses. https://status.mstdn.ca/incident/272175 Sun, 15 Oct 2023 07:00:00 -0000 https://status.mstdn.ca/incident/272175#fe05ac123d2e2229b8993adcf59982862e0b50bacc78a34feaea62d4c1c7687c Maintenance completed https://status.mstdn.ca/incident/272175 Sun, 15 Oct 2023 04:00:00 -0000 https://status.mstdn.ca/incident/272175#b2a4a3d4e802918fe509b5befc37c7f6b8d0ed11eff50c23a0e4d4323bef194c Mastodon Canada will be taking a downtime during the window indicated to perform a series of tests to confirm that our databases are ready to fail-over to the new lower cost database servers. This test will require that the instance take a downtime lasting up to two hours. The instance will be inaccessible during this time. A four hour window is being allotted for preparation and restoration tasks. https://status.mstdn.ca/incident/272175 Sun, 15 Oct 2023 04:00:00 -0000 https://status.mstdn.ca/incident/272175#b2a4a3d4e802918fe509b5befc37c7f6b8d0ed11eff50c23a0e4d4323bef194c Mastodon Canada will be taking a downtime during the window indicated to perform a series of tests to confirm that our databases are ready to fail-over to the new lower cost database servers. This test will require that the instance take a downtime lasting up to two hours. The instance will be inaccessible during this time. A four hour window is being allotted for preparation and restoration tasks. https://status.mstdn.ca/incident/261045 Wed, 27 Sep 2023 08:00:00 -0000 https://status.mstdn.ca/incident/261045#85ed9a258ed7ea92f3a2dce57395bd15d8ace0a45134b1636d117e93061bc0bf Maintenance completed https://status.mstdn.ca/incident/261045 Wed, 27 Sep 2023 04:00:00 -0000 https://status.mstdn.ca/incident/261045#5a1e9eaf6ffe63db7e5725d5b2c557a02fb5552e8d1e7f440ec2d58813bee2e3 Please be advised Mastodon Canada will be undergoing software security patching and implementing infrastructure changes during this scheduled maintenance. No downtime is expected however we could experience a period of instability. This is also another step towards moving our instance from DigitalOcean to OVH. https://status.mstdn.ca/incident/261045 Wed, 27 Sep 2023 04:00:00 -0000 https://status.mstdn.ca/incident/261045#5a1e9eaf6ffe63db7e5725d5b2c557a02fb5552e8d1e7f440ec2d58813bee2e3 Please be advised Mastodon Canada will be undergoing software security patching and implementing infrastructure changes during this scheduled maintenance. No downtime is expected however we could experience a period of instability. This is also another step towards moving our instance from DigitalOcean to OVH. https://status.mstdn.ca/incident/239502 Tue, 01 Aug 2023 04:14:00 -0000 https://status.mstdn.ca/incident/239502#661a2394d1f9a7bdb1c2a3f86bc6a0e9c8fd1e0e5a31bfbb0cfbe2d54656a7b0 We identified the root cause of the problem and were able to remove the problematic records. This has cleared up the notification delay. https://status.mstdn.ca/incident/239502 Mon, 31 Jul 2023 20:10:00 -0000 https://status.mstdn.ca/incident/239502#fa4ddf88ad75bb0c22376b404ad71841d3aaca2e22759874aa7679153750d890 Notifications are still delayed by about one hour, which is significantly better than where we were last night. We'll continue to monitor the situation. https://status.mstdn.ca/incident/239502 Mon, 31 Jul 2023 05:32:00 -0000 https://status.mstdn.ca/incident/239502#4b60623209f633b0ff61c30b274e466ebef018e4216c8a1644dcd8b1881df6d2 You may have noticed that you aren't getting timely notifications from mstdn.ca today. A third-party push notification service that many popular apps use went down yesterday and caused our notification queues to pile up. Additional resources have been dispatched to work to catch up on the notification queues. https://status.mstdn.ca/incident/236001 Sun, 23 Jul 2023 07:38:30 -0000 https://status.mstdn.ca/incident/236001#9592476fe8cde1eee3383c3f7cb87c42bf80b81ae949d51817bf0359e7db0b37 Maintenance completed https://status.mstdn.ca/incident/236001 Sun, 23 Jul 2023 05:30:00 -0000 https://status.mstdn.ca/incident/236001#e7eed1428c08caeed7bf160da0f83eaeb650e8f64ada2748c782a9f2e72b8a4e System and security upgrades will be performed across the entire system, including operating system patches and a Mastodon software upgrade. Service may be degraded during this window. https://status.mstdn.ca/incident/236001 Sun, 23 Jul 2023 05:30:00 -0000 https://status.mstdn.ca/incident/236001#e7eed1428c08caeed7bf160da0f83eaeb650e8f64ada2748c782a9f2e72b8a4e System and security upgrades will be performed across the entire system, including operating system patches and a Mastodon software upgrade. Service may be degraded during this window. https://status.mstdn.ca/incident/230137 Sat, 08 Jul 2023 01:21:10 -0000 https://status.mstdn.ca/incident/230137#27fa8e66928125da5ff89d77d34b83ed82d913b3940b46da7b904827fc0fe4d6 Maintenance completed https://status.mstdn.ca/incident/230137 Sat, 08 Jul 2023 00:00:00 -0000 https://status.mstdn.ca/incident/230137#f2c4ce3e7281f3d33fb934ae85ed7b3baf5d5c3906bdf4e410964ca1c5865d8d Upgrades are underway to mitigate the RootToot vulnerability. https://status.mstdn.ca/incident/230137 Sat, 08 Jul 2023 00:00:00 -0000 https://status.mstdn.ca/incident/230137#f2c4ce3e7281f3d33fb934ae85ed7b3baf5d5c3906bdf4e410964ca1c5865d8d Upgrades are underway to mitigate the RootToot vulnerability. https://status.mstdn.ca/incident/207626 Tue, 16 May 2023 08:00:00 -0000 https://status.mstdn.ca/incident/207626#943b2ae5c8347d2fba9a14d98300629eba09e1b0ab814920c243b82e63aa059a Maintenance completed https://status.mstdn.ca/incident/207626 Tue, 16 May 2023 04:30:00 -0000 https://status.mstdn.ca/incident/207626#e9cdee6cd3b52790fbf40c8b470118336b16ad114c16c9acd9751ca96ae2b562 System and security upgrades will be performed across the entire system, including operating system patches and a Mastodon upgrade. No downtime is expected however service may be degraded. https://status.mstdn.ca/incident/207626 Tue, 16 May 2023 04:30:00 -0000 https://status.mstdn.ca/incident/207626#e9cdee6cd3b52790fbf40c8b470118336b16ad114c16c9acd9751ca96ae2b562 System and security upgrades will be performed across the entire system, including operating system patches and a Mastodon upgrade. No downtime is expected however service may be degraded. https://status.mstdn.ca/incident/142125 Tue, 06 Dec 2022 11:33:00 -0000 https://status.mstdn.ca/incident/142125#d1f48fe27d15282e35f17e1eea43b388452719ba345dc9b0eb3a5219ed308e9b Our ISP's Engineering team has confirmed the resolution of the issue with networking in our data centre. https://status.mstdn.ca/incident/142125 Tue, 06 Dec 2022 05:43:00 -0000 https://status.mstdn.ca/incident/142125#93528415cfda203b6f879a7dad9fc532170e11d8aab1828a9e66ca2cc1267ff6 DigitalOcean has alerted us that their Engineering team has identified an issue with networking in our datacenter. Some users may experience packet loss and increased latency as they work to mitigate the issue. https://status.mstdn.ca/incident/139010 Fri, 25 Nov 2022 09:00:00 -0000 https://status.mstdn.ca/incident/139010#c7cb4a6d41dd7b049c7efc3aa2c65b8f82e8d6e6397362dc9ddc946642c48f35 Maintenance completed https://status.mstdn.ca/incident/139010 Fri, 25 Nov 2022 06:30:00 -0000 https://status.mstdn.ca/incident/139010#9931c32cf5fb06a15a07905b94c59f8f4cf70244ddb3e956ee6d2bfb476f6770 Please be advised that another round of infrastructure changes is scheduled for tonight and some downtime is expected.