Mark T. Tomczak on Nostr: Ah, you've just inspired me to bother to look at my robots.txt file. ... looks like ...
Ah, you've just inspired me to bother to look at my robots.txt file.
... looks like Mastodon node doesn't disallow the Google crawlers by default, so I assume it's not unexpected your content would show up in Google's search engine? Though one thing I've never stopped to wonder: if you *do* disallow Google's spider, but your posts are imaged to someone else's node because someone on their node follows you, can the spider pick up your posts from there? This post on your server also has [https://mastodon.fixermark.com/@lauren@mastodon.laurenweinstein.org/112649798768708524](A URL on my server), so I'm *pretty* sure the spider could see your post via my node even if you disallow it on your node.
... looks like Mastodon node doesn't disallow the Google crawlers by default, so I assume it's not unexpected your content would show up in Google's search engine? Though one thing I've never stopped to wonder: if you *do* disallow Google's spider, but your posts are imaged to someone else's node because someone on their node follows you, can the spider pick up your posts from there? This post on your server also has [https://mastodon.fixermark.com/@lauren@mastodon.laurenweinstein.org/112649798768708524](A URL on my server), so I'm *pretty* sure the spider could see your post via my node even if you disallow it on your node.