On Tue, Jan 27, 2026 at 09:21:59AM +0100, Tom Szilagyi said:
>Most humans use browsers with JS; most bots (including all the LLM
>scrapers) do not bother. So this is transparent for actual humans
>browsing around. For my case, this approach solved the problem
>completely (for now).
I have found more and more that AI scrapers tend to try with a
simplistic bot first and then come back with what looks like headless
Chrome. I've seen lots of them that are fully executing JavaScript just
like most modern search engine crawlers. I've taken (for better or
worse) to flat out dropping traffic from a number of ASNs at the edge
along with everything else.
--
Please direct replies to the list.
No comments:
Post a Comment