Rob Carlson on Nostr: Here's some pretty clear evidence that at least one set of web crawlers are ...
Published at
2025-01-15 16:12:49Event JSON
{
"id": "3e6b7e2eb6815c464b7c4abdf62312dd9305c54d429b7492bca817ccb4e42e28",
"pubkey": "6b9b4d5f1cdd4a5381f778caac8c80df2244a3407b1b74123b935d0156cd7886",
"created_at": 1736957569,
"kind": 1,
"tags": [
[
"proxy",
"https://epistolary.org/users/vees/statuses/113833251286689539",
"activitypub"
]
],
"content": "Here's some pretty clear evidence that at least one set of web crawlers are disguising their user agent as a human and sidestepping HTTP 429 on shared servers by immediately switching IP addresses instead of falling back and waiting. https://gist.github.com/vees/7941c368230b708954fefedbf801358c",
"sig": "8fa6622febebc47bebcae7f45ac298a68b12d78ecaf215e259164df7bd2f5342f1e6ff6770259db9ebe735236aaaa86143c20dd0006d329683a2d804ef418925"
}
