Dr. Eric J. Fielding, PhD on Nostr: npub1lrvmc…jtth9 Maybe the news sites give Google web crawlers special access ...

npub1lrvmc88ck63qpt734w85t5pnxchncvam0p9k9mueuezqz0grkn6s8jtth9 (npub1lrv…tth9) Maybe the news sites give Google web crawlers special access without the paywall? News sites want people to find their content. It seems that Twitter is specifically trying to block Google and others from scraping any information, so I am sure they are blocked.