dansup on Nostr: Adding robots.txt support to FediDB, if you want to block FediDB from crawling your ...
Adding robots.txt support to FediDB, if you want to block FediDB from crawling your instance, please add a disallow rule to the ./well-known endpoint like GtS is doing.
https://goblin.technology/robots.txtWe will be dropping the Popular Accounts section as those are more harder to support, and I will re-enable them once I can get around to it.
Published at
2025-02-07 06:30:18Event JSON
{
"id": "f041007e8ea23cd0f51a226847dafe3f06f524ba77e722f9790c18f829ee65eb",
"pubkey": "fce95231cd584e791f1f5d977ceac1ef6edb3d3a7a29ada5a657979836cbcb1f",
"created_at": 1738909818,
"kind": 1,
"tags": [
[
"imeta",
"url https://files.mastodon.social/media_attachments/files/113/961/183/903/835/559/original/66ee250ac9e9ffc5.png",
"m image/png",
"dim 570x132",
"blurhash U12P;xyEQ?tx*Fa7V@adMJV=pFVzXup8jco#"
],
[
"proxy",
"https://mastodon.social/users/dansup/statuses/113961193872712501",
"activitypub"
]
],
"content": "Adding robots.txt support to FediDB, if you want to block FediDB from crawling your instance, please add a disallow rule to the ./well-known endpoint like GtS is doing.\n\nhttps://goblin.technology/robots.txt\n\nWe will be dropping the Popular Accounts section as those are more harder to support, and I will re-enable them once I can get around to it.\n\nhttps://files.mastodon.social/media_attachments/files/113/961/183/903/835/559/original/66ee250ac9e9ffc5.png",
"sig": "e1941a3e107462eb35356e15f654b10ba585124d1a8bb71353d6d0e4751e869c45f5b7f7d075c1c86e467a1a003c75451451410f8fe085cd7d7214b5dd67d6c0"
}