
Sinkhole on State Highway 23 near Hatton North Dakota
A website owner frustrated with the high rate of scraping that AI web crawlers were conducting on his site — 30 million hits from Facebook’s crawler alone — developed Nepenthes, a program that uses the cybersecurity tactic of tarpitting. The technique returns bogus results to scrapers, sending them on meandering and pointless ventures down an infinite maze with no exit link, fed useless data designed to poison AI models in the process, all for not respecting robots.txt. Tarpits were originally used to waste spammer time, but the anti-scraper program, Nepenthes, and another one it inspired, Iocaine, are damn effective, with the latter instantly killing 94 percent of bot traffic fired at a site. When the world hands you minotaurs, make labyrinths.
Comments