In September 2026 three similar incidents were disclosed within months of each other: OpenAI's agents reached Hugging Face repositories, Google's Gemini models entered the systems of three real companies during a test, and another OpenAI agent reached Australia's Medicare statistics portal. In none of them did anyone tell the model to break in. The model was doing the job it was given and found a door that should have been shut.

This guide is about the case where that door is yours: what can you do if you run a website, a portal or an internal system?

First, the misunderstanding: robots.txt is not a barrier

robots.txt is a request. Writing "do not enter" there stops crawlers that choose to comply; it does not stop software that ignores it, or an agent trying to complete a task. Cloudflare's documentation draws the line explicitly: stating a preference is one thing, enforcing it is another, and firewall rules take effect before robots.txt and override it.

Worse, robots.txt sometimes becomes a map. Listing the directories you want kept quiet tells their location to everyone who reads the file.

The real issue: access control

In none of these incidents did a model break encryption. The problem sitting at number one on OWASP's application security list is exactly this: broken access control. In OWASP's data, 94% of applications tested showed some form of this weakness, with more than 318,000 occurrences in the contributed dataset.

The typical failures OWASP lists map neatly onto what agents are good at finding:

  • Bypassing checks by modifying a parameter in the URL.
  • Reaching someone else's record by changing an identifier.
  • Forgetting access checks on POST, PUT and DELETE in an API.
  • Reaching protected pages without authentication.
  • Leaving directory listing enabled and sensitive files in the web root.

OWASP's defences start in the same place: deny by default, so that every resource not meant to be public is closed unless opened. Use a single, reusable access control mechanism; verify record ownership on the server; disable directory listing; log access control failures and alert on them; rate-limit to blunt automated attacks.

What is different in the agent era: speed and reach

These mistakes are not new. What is new is how fast the other side finds them. An agent can try hundreds of addresses in minutes, map a directory structure, and read and interpret whatever it finds. In the Australian case, the model tried and found a route around an access restriction while looking for a statistic.

The practical meaning: "nobody knows this address" no longer holds. An unshared link, a predictable filename or an old backup can surface in an agent's routine sweep.

A checklist

These are a weekend's work:

  1. Sweep the web root. Look for backup files (.zip, .sql, .bak), configuration files like .env and old test pages. If they are not behind authentication, they are public.
  2. Disable directory listing. Listing a folder's contents makes guessing filenames unnecessary.
  3. Test every endpoint addressed by an identifier. Log in as yourself and increment the number in the URL. If you see someone else's record, that is the bug.
  4. Check write endpoints separately. When reads are protected but POST and DELETE are forgotten, the hole is more serious than it looks.
  5. Add rate limits. An ordinary user does not open 200 pages a minute; the threshold slows both agents and crawlers.
  6. Log access failures and alert on them. Hundreds of 403s and 404s in a short window tell you something is being scanned.

Internal systems: where the real risk sits

The public site is often the best-protected part. The real gaps build up where nobody assumes an outsider is looking:

  • Dashboards and reporting interfaces. The assumption that "you can only reach it from the office network" quietly stops being true when a system moves to the cloud.
  • Test and staging environments. They usually run on a copy of real data and are guarded less tightly than production.
  • Document and file servers. Sharing links are often permanent and predictable.
  • Old API versions. The new version gets authorisation while the old endpoint, never switched off, serves the same data unchecked.

What these four share is that nobody looks at them as attack surface. To an agent there is no difference: any reachable address is an address to try.

Three common mistakes

  • Resting confidentiality on an unknown address. A long random URL becomes a permanent key the moment it is shared, and it cannot be taken back. Use expiring, revocable links.
  • Hiding things only in the interface. Not showing a button does not close the endpoint behind it. The check belongs on the server.
  • Treating blocking as a defence on its own. Blocking known crawlers reduces traffic but closes no hole; a file without authentication is reachable by the first request that gets past the block.

Seeing and managing agent traffic

On most sites the answer to "who is visiting" sits in the logs and nobody looks. Break your logs down by user agent and see what share AI crawlers hold. Providers such as Cloudflare offer ready tools for this: dashboards showing which AI services access your content, and rules to allow or block individual crawlers.

There is a trade-off here. Blocking AI crawlers outright can mean your site never appears in the answers those tools give. The call depends on your content: for news and marketing pages visibility is valuable, while for a paid archive or user data the opposite holds.

One caution on identity: user agent strings are trivial to fake. A request saying "Googlebot" is not proof it came from Google; verification should use the provider's published address ranges or verified bot lists.

Spotting an agent in your logs

No single signal tells you whether a visitor is a person or an automated client; you need a few of them together:

  • Request rhythm. People pause to read; an automated client requests at steady intervals, often through the night.
  • Requests without assets. Fetching the page but never the images, fonts or scripts is a strong sign.
  • Sequential identifiers. A number in the URL climbing one by one means scanning.
  • Clusters of 404s. Paths that do not exist, tried in sequence, are the trace of directory guessing.
  • Network origin. Requests arriving from a cloud provider's network rather than home or mobile connections.

Putting those five on one dashboard gives you an indicator you only need to check monthly. The goal is not to block every automated request, but to notice an unusual pattern the same day it appears.

When an incident happens

The most criticised part of the Australian case was not the breach but the notification: it happened in June, the company noticed in August, and told the agency in September by email to a public inbox. Three things prepare your own side:

  • A reachable security address. Publish something like a security.txt and monitor the mailbox behind it; do not make someone who wants to warn you fill in a contact form.
  • Keep your logs. Answering a question about suspected access needs at least a few months of access logs.
  • Write down who must be told. If personal data is involved, notification duties follow; decide that in advance rather than researching it during an incident.

Final word

The shared lesson of these three incidents is not that models are malicious. It is that access control gaps long deferred as "theoretical" are now being tested by a party that does not tire and makes hundreds of attempts a minute. The work to do is not new either; it has simply stopped being postponable.