Alexandre Dulaunoy on Nostr: Did you know? The Facebook crawler doesn't consider most HTTP return codes except for ...
Did you know? The Facebook crawler doesn't consider most HTTP return codes except for 403!
If a page returns 403, the Meta crawler won’t access it. Other status codes, like 404 or 451, won’t stop it from attempting to scrape your page’s content. The crawler will continue and won´t apply any delay.
#crawler #meta
If a page returns 403, the Meta crawler won’t access it. Other status codes, like 404 or 451, won’t stop it from attempting to scrape your page’s content. The crawler will continue and won´t apply any delay.
#crawler #meta