GrapheneOS on Nostr: Our Mastodon server stopped working after around 22:37 UTC time yesterday. It appears ...
Our Mastodon server stopped working after around 22:37 UTC time yesterday. It appears Mastodon spun out of control consuming lots of CPU, disk and memory. It locked up the server with a massive amount of I/O. There isn't a lot of information about what happened since logging stopped persisting to storage shortly after it started. It was quickly detected by our service monitoring but we just weren't around to fix it for 7 hours. It's working again after a server restart but nothing was changed.
Published at
2025-02-12 08:18:38Event JSON
{
"id": "f6a6dfbfc0b69093bf669521021d7fea84d50df7e59dc10be06121ab77c1be40",
"pubkey": "5468bceeb74ce35cb4173dcc9974bddac9e894a74bf3d44f9ca8b7554605c9ed",
"created_at": 1739348318,
"kind": 1,
"tags": [
[
"proxy",
"https://grapheneos.social/users/GrapheneOS/statuses/113989931398927308",
"activitypub"
]
],
"content": "Our Mastodon server stopped working after around 22:37 UTC time yesterday. It appears Mastodon spun out of control consuming lots of CPU, disk and memory. It locked up the server with a massive amount of I/O. There isn't a lot of information about what happened since logging stopped persisting to storage shortly after it started. It was quickly detected by our service monitoring but we just weren't around to fix it for 7 hours. It's working again after a server restart but nothing was changed.",
"sig": "ad9a02007f1395941341042191d4df3974bb98014995aabf7d1b16d22416750d442926cc48b56d912b3f7b72af1163c0d71877effab5ba603d150c928c812c24"
}
