Protecting your site from AI agents: robots.txt is not enough
AI agents can cross access boundaries while doing the task they were given. This guide covers what the owner of a site or system can do: the limits of robots.txt, access control, seeing agent traffic and handling an incident.