AI独角兽视共识于无物，互联网公地悲剧即将上演

Core Insights - The AI industry is facing a "data wall" as predicted by Epoch AI, which suggests that by 2028, all high-quality text data on the internet will be exhausted, leading to a struggle between AI companies seeking data and data owners [1] Group 1: Company Actions and Reactions - Cloudflare accused AI search unicorn Perplexity of violating website data scraping rules by ignoring the robots.txt file that prohibits AI crawlers from accessing certain content [2][4] - Perplexity allegedly disguised its crawlers as Chrome user agents to bypass website restrictions, prompting Cloudflare to remove Perplexity from its verified bot list [4][9] - Perplexity's spokesperson denied Cloudflare's claims, suggesting that Cloudflare's actions were self-serving and aimed at promoting its own services [4][8] Group 2: Industry Standards and Implications - The robots.txt file is a foundational element of internet standards, indicating which content is off-limits to crawlers, thus preserving bandwidth and server resources for website owners [11] - The disregard for established norms by companies like Perplexity could lead to a "tragedy of the commons," where excessive use of internet resources discourages content creators from sharing their work [13][14] - Cloudflare's introduction of a Pay Per Crawl platform indicates a potential monetization strategy in response to the challenges posed by AI crawlers, highlighting the ongoing conflict in the industry [9]