Migrating this site to EmDash
This site has changed CMS more times than I can count since 2003. This time I moved to EmDash on Cloudflare, and the reason is that most of my building now happens alongside an AI agent.
Read moreHow to Block OpenAI’s Web Crawler from Crawling Your Site
Roger StringerAugust 10, 2023
1 min read
Just as you can use robots.txt to block search engines and various bots from crawling your site, you can do the same with OpenAI's GPTBot.
GPTBot is OpenAI’s web crawler and can be identified by the following user agent and string.
User agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)To block GPTBot from crawling your site entirely, you can add the following to your robots.txt file:
User-agent: GPTBot
Disallow: /If you want to allow GTPBot to crawl certain areas of your site, but block it from other areas, you can do this:
User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/Tagged In:
Do you like my content?
Sponsor Me On GithubKeep reading
Get Butters: A Quick Events Dashboard
I wanted a simple dashboard that showed me events as they happened, so I wrote a quick little Astro app. It grew into Get Butters: a real-time event feed with a REST API, a CLI, insight cards and push alerts.
Read more
The Agentic OS in Practice: Running Multiple Agents Without Losing Control
Once four agents can edit the same repo, architecture stops being the hard problem and coordination starts. The five roles I run and what each one is forbidden to touch, why every handoff goes through a file, a worktree script so agents stop racing, and the failure modes worth naming.
Read more