Traditional search engines are no longer the only entry point to a site. Conversational agents, AI-powered browsers, autonomous crawlers: the intermediaries between web content and its reader are multiplying, each applying its own extraction rules. For online media and businesses, keeping up with web and digital news is no longer enough. It is essential to understand how these new technological layers redistribute visibility, traffic, and security risks.
AI Robots and Crawling: The New Segmentation Imposed by Cloudflare
Cloudflare has changed its policy regarding AI robots by blocking training and exploration crawlers by default on new domains. The distinction is no longer between “allowed bot” and “blocked bot,” but between several categories of robots with different purposes: model training, indexing for generative engines, autonomous agents performing tasks.
This segmentation changes the game for publishers. A classic robots.txt file is no longer sufficient to control who accesses what. We observe that sites that have not reviewed their configuration allow AI agents while blocking legitimate crawlers, or vice versa.
Following the news on Blognet News allows you to spot these technical developments as soon as they are announced, before they affect organic traffic. The topic goes beyond simple monitoring: it involves adapting the access layer of your site to a fragmented ecosystem of robots.
- Check server logs for the share of traffic generated by AI user agents (GPTBot, ClaudeBot, PerplexityBot, among others) and their frequency of access
- Set distinct rules for each category of robot, rather than a blanket block or global access
- Monitor updates from CDNs and application firewalls, which now include continuously maintained lists of AI robots
SEO in the Face of Generative Engines: What Changes in Acquisition
Generative responses capture an increasing share of clicks that previously led directly to a publisher’s site. Google AI Mode, integrated responses from Perplexity, Bing Copilot summaries: content is consumed without the user visiting the source.
For a web media outlet, this forces a rethink of the very notion of visibility. Being cited in a generated response does not produce the same return as a clicked link. The effective click-through rate on organic results decreases when an AI response satisfies the query directly in the engine’s interface.
Structuring Content for Agents, Not Just for SERPs
Structured data (schema.org) takes on an additional function. It is no longer used solely to obtain a rich snippet, but to provide language models with usable context. Precise markup of the type of article, publication date, author, and mentioned entities increases the likelihood of being selected as a source by a generative engine.
We recommend treating each page as a semantic API readable by an agent. Explicit titles, standalone paragraphs (each block answers a question), alt attributes filled in for images, structured breadcrumbs. These practices already existed in technical SEO, but their relative weight is increasing in the face of AI agents.
AI Agents in Online Commerce: Alert from the Competition Authority
The Competition Authority has warned about the specific risks of AI agents in online commerce. These agents, capable of comparing offers, placing orders, and negotiating prices without human intervention, are becoming new entry points to merchant sites.
The issue is not limited to competition. An AI agent interacting with an e-commerce site raises concrete security questions: authentication of requests, detection of automated scraping disguised as user navigation, management of sessions and carts created by bots.
Security of AI-Automated Workflows
Platforms oriented towards “agentic AI” no longer just generate text. They plan, modify, and manage websites end-to-end, directly within CMSs. An agent can publish an article, modify a product sheet, adjust a price.
Every automated action in a CMS must be audited as a standalone user action. Activity logs must distinguish human modifications from agent modifications, with granular permission levels. Without this traceability, a security incident caused by a poorly configured agent becomes impossible to diagnose.
- Assign dedicated API tokens to each AI agent, with permissions restricted to the bare minimum
- Enable detailed logging of CMS actions to isolate automated modifications
- Set up alerts for abnormal behaviors: mass publishing, price changes outside defined ranges, content deletion
- Regularly test intrusion scenarios using simulated agents to identify vulnerabilities before a malicious actor can exploit them
AI Browsers and the End of ChatGPT Atlas: Lessons for Web Publishers
OpenAI announced the discontinuation of ChatGPT Atlas, its AI-powered browser, less than a year after its launch. The product, deemed too minimalistic, never posed a threat to Chrome. This failure illustrates a point often underestimated: an AI browser does not replace a traditional browser, it complements or parasitizes it.
For publishers, the lesson is about dependence on distribution channels. A site optimized exclusively for an AI browser or engine is exposed to a sudden loss of traffic if that channel disappears. Diversifying acquisition sources remains the best protection, including towards RSS feeds, newsletters, and podcasts, which escape algorithmic intermediation.
The pace of appearance and disappearance of these tools is accelerating. Adapting your digital strategy to each new AI product is not sustainable. However, maintaining a clean technical architecture, rigorous semantic markup, and well-segmented access rules provides a foundation that withstands surface changes. It is on this technical layer that the mastery of online visibility now hinges.