Hacker News
Welcome to LWN.net
The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!
By Jonathan Corbet
July 10, 2026
Our article "Fighting the AI scraper bot
scourge", published in early 2025, discussed the problem of widespread
scraping of web sites in search of training data for large language models
and related projects. This activity overwhelms sites with traffic. Over a
year after that article is published, the problem is still growing. The
hammering of sites by shadowy actors has reached new heights, and the open
web is becoming increasingly difficult to maintain. Where is this traffic
coming from, and what can be done about it?
Residential proxies
As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term "residential proxies" is used to describe systems that are used in this way.
There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.
More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
The second sort of operator works more overtly, pretending to a degree of legitimacy and offering "ethically sourced" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a "free" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.
There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.
Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt. They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.
What isn't clear is who is using the residential proxies; somebody is paying them to run these attacks on web sites. There is no evidence (that I am aware of) that the frontier-model companies are using those networks. If were to turn out that they are doing so, though, the increase in global astonishment would barely register. Those companies are feeding their models somehow, they are not forthcoming about how they get their training data, and they have not distinguished themselves with their level of respect toward content creators — or toward anybody who might have concerns about their operations.
For every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organizations (to the extent that they are distinct from governments) probably also want to have their own models. These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.
Defending the open Internet
In response to all of this, web-site operators have been scrambling to defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by requiring a proof of work, is now widespread. Other sites use commercial services, which sometimes make themselves known with a "prove you are human" button. Or sites force users to pick out squares containing streetlights (but only those with LED bulbs), place puzzle pieces, or hum a song while holding down the space bar. Many site features have been placed behind login gates or paywalls. Some sites attempt to actively poison the data sent to scrapers with tools like iocaine.
Both the need to set up and maintain these mechanisms, and the requirement that users cope with them to access a web site, constitute a heavy tax placed on the world as a whole by scrapers and those who pay them.
Recently, LWN was subjected what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.
What we can say is that we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.
We have aggressively optimized parts of the site, and found ways to minimize expensive operations during times when the site is under attack. Anonymous readers may occasionally encounter one of those measures; logged-in users will not. Amusingly, the response time when the site is under attack is often better than during the calm times, when the defensive measures are dormant. We have learned better than to think that the problem is solved, though; consideration must be given to our next steps once the current measures are no longer effective.
On July 2, Google announced that it had, in coordination with the US Federal Bureau of Investigation and others, taken down a residential-proxy network called "NetNut". For the time being, that action would, indeed, seem to have succeeded in reducing the level of scraper attacks somewhat. Experience shows, though, that this welcome peace will only last so long. Google takes pains to point out that its Play Store will now check for NetNut-infected apps, but all of the major vendors are silent on the topic of why it is so easy to put apps with residential-proxy functionality into their app stores.
It would be good to find a more lasting solution before the entire Internet
is driven behind defensive walls, and the open network that inspired so
much creativity is lost. The industry that is driving these attacks seems
entirely at ease with turning independent web sites into smoking craters
after having pillaged their contents — an attitude that extends to the
planet and its economies as well. Some of us, though, object to that idea
and will fight against it. Someday, with luck, the world as a whole will
decide to hold the companies behind large language models and related
technologies to a minimal ethical standard. Until then, though, this
behavior will continue, and we will have no choice but to defend ourselves
against it.
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 16:25 UTC (Fri) by farnz (subscriber, #17727) [Link]
If you're using Google Play Services, turn on Play Protect and have that scan every so often - this will catch any apps that Google has found contain one of these proxies.Other than that, you can ask Android to tell you how much mobile data an app used; go to "Settings", then "Apps", and it's in the "App Info" screen for individual apps. You can also disable "Background data" if you're suspicious of an app - that stops it using data when you don't have it open and are on mobile; IIRC, this also covers WiFi networks set as "metered", but ICBW on that one.
How to avoid running a residential proxy (especially on Android phones)?
Posted Jul 10, 2026 20:26 UTC (Fri) by mirabilos (subscriber, #84359) [Link]
TEMU is rumoured to contain one. AFAIHH it just runs in the foreground while the device users browse its marketplace.
Note this is hearsay. I have not installed that äpp nor sniffed its traffic myself. But it was the first one where I heard about what is now called residential proxies, via “trojaned” äpps, and from multiple sources.
I’m sure the “link fetchers” from GAFAM äpps can also occasionally do that…
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 16:08 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (2 responses)
Hi Jon,
I think that many of us here are totally aware of the problem these bots and proxies are causing to web sites like this one, and the difficulties in fighting them. At least I can say that I haven't noticed anything abnormal on the site here, so your actions were fine from the user experience perspective. Thanks for this!
You're right, it's important never to publicly explain the counter-measures that you apply. Very often some are extremely simple and effective (some easy tricks I've deployed 9 months ago that I imagined would only last one week are still working fine). Also they're often very specific to the site and would hardly adapt to other ones (except for the main principle), so there's little to share by explaining everyone how your specific site is fighting these.
I noticed a 30% drop of traffic on July 2nd, after a 50% one on June 25th that I couldn't explain (mostly attributed to user-agent "sleepbot"). So yes, it seems that such networks are progressively getting dismantled, probably to re-appear somewhere else soon, given that infected browsers (and their unsuspecting users) are just waiting for another C&C to take care of them :-/
I must confess I'm a bit worried about the risk of losing a lot of legit content indexing on the net in the coming years due to installed counter-measures against non-humans. If sites cannot be found via search engines it will become a problem. All this due to AI startups racing in training their own models (or variants).
Maybe it would work better to set up a static central registry of the whole internet's contents that could be scraped by such companies as much as they want without killing small web sites. It could still take a lot of time before we start to see something like this happen though.
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 17:15 UTC (Fri) by daroc (editor, #160859) [Link] (1 responses)
The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/
The idea is that you contribute resources to their project, they have _one_ scraper that downloads web content in a polite way (respecting rate limits and robots.txt), and then anyone who wants to can make use of the scraped content without having to re-scrape it.
There are some problems with the approach, but I'm generally pretty happy when I see Common Crawl go by in the server logs because I know that's a bunch of unrelated projects that don't need to send us multiple requests.
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 17:46 UTC (Fri) by wtarreau (subscriber, #51152) [Link]
> The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/
Oh, thanks for the link, I wasn't aware. I'll try to make sure not to block that one!
Preventing this at the app store level is hard
Posted Jul 10, 2026 20:42 UTC (Fri) by roc (subscriber, #30627) [Link]
> all of the major vendors are silent on the topic of why it is so easy to put apps with residential-proxy functionality into their app stores.
Google's rules already forbid non-user-authorized "residential proxies": https://support.google.com/googleplay/android-developer/a...
But if the user gives genuine consent to it, should they still be banned? That's a tough call.
Of course developers can and do violate those rules. Then the problem is that it's not really possible in general to detect that code is going to break those rules just by inspecting it. So how do you detect violations other than by people reporting them and then shutting them down after the fact? That's roughly what happens now and it's better than nothing, but still whack-a-mole.