We’re having problems with bots hammering the site. For example since this morning we’ve got 65 thousand requests for bibliography items (and it isn’t noon yet). As we have only 700 bibliography entries, this means there are more than 90 bots each requesting the entire bibliography — some kind of botnet that’s going wild.<\/p>\n
While these idiotic bibliography requests are a new problem, heavy traffic due to bots is not a new problem. It’s frustrating as any smart web-crawler would understand that all the information could be obtained by downloading the entire stacks project from github in one fell swoop. An example is 36K requests from openai\/bot (related to ChatGPT) just today; more than 10% of our traffic.<\/p>\n
Meanwhile I apologize to any humans still reading the Stacks Project!<\/p>\n
PS: We had to temporarily block access to the bibliography pages.<\/p>\n","protected":false},"excerpt":{"rendered":"
We’re having problems with bots hammering the site. For example since this morning we’ve got 65 thousand requests for bibliography items (and it isn’t noon yet). As we have only 700 bibliography entries, this means there are more than 90 … Continue reading