Creepy crawlies

תקציר מקורי באנגלית<p><strong><a href="https://people.kernel.org/monsieuricon/creepy-crawlies">Creepy crawlies</a></strong></p> Konstantin Ryabitsev discusses how bad the "background radiation" of abusive crawlers has become from the perspective of <a href="https://git.kernel.org/">git.kernel.org</a>, the official Git repository for the Linux kernel:</p> <blockquote> <p>TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.</p> </blockquote> <p>I worry about this a lot from the perspective of Datasette, which serves a huge number of crawlable web pages. <p><small></small>Via <a href="https://news.ycombinat