https://noted.jsrowe.com/tags/robots.txt/Recent content in Robots.txt on Duly NotedHugoen-usMon, 07 Apr 2025 02:27:26 +0000https://noted.jsrowe.com/robots-all-the-way-down/Mon, 07 Apr 2025 02:27:26 +0000https://noted.jsrowe.com/robots-all-the-way-down/<blockquote> <p>we found out that at least 65% of this resource-consuming traffic we get for the website is coming from bots, a disproportionate amount given the overall pageviews from bots are about 35% of the total. This high usage is also causing constant disruption for our Site Reliability team, who has to block overwhelming traffic from such crawlers before it causes issues for our readers.</p></blockquote> <p>I do wonder about the future of internet. Crawling is nothing new, its how google built its original search engine. But if 65% of traffic is robots where are the humans?</p>https://noted.jsrowe.com/example-of-ai-not-honoring-robots.txt/Sun, 30 Mar 2025 02:46:52 +0000https://noted.jsrowe.com/example-of-ai-not-honoring-robots.txt/<blockquote> <p>The AI scraper (I can only assume thats what they are) scourge continued, and intensified in the last week. This time they were hitting pagure.io really quite hard.</p></blockquote> <p>Robots.txt is only honor system. One of many postings calling it out.</p> <hr> <p>_{Quote Citation: <cite>Kevin, &ldquo;Mid March infra bits 2025&rdquo;, 2025-03-15 17:52, <a href="https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/"><a href="https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/">https://www.scrye.com/blogs/nirik/posts/2025/03/15/mid-march-infra-bits-2025/</a></a></cite>}</p>