Meta's AI Crawler Crashing My DB, and What I Did About It

On Friday I got an email from Sentry to say I'd used 81% of free monthly errors. Usero doesn't have that many users, so I was annoyed and suspicious (of bots).

The top issue was D1_ERROR: D1 DB is overloaded. Requests queued for too long. 4,227 events in 14 days, about 80% of everything Sentry had caught. It was all one user, one IP, hitting /__manifest. That's the endpoint React Router uses to prefetch the route manifest for the links on a page.

So who is it?

The IP was 2a06:98c0:3600::103. It looks like a datacenter, so I looked it up and found ASN 13335. That's Cloudflare. Usero runs on Cloudflare Workers, so the "user" Sentry saw was the edge itself, and "one user" meant everyone arriving through it. A Sentry user IP in Cloudflare's range isn't a person.

I needed the real client IP, so went looking for logs. I spent an hour trying to get Cloudflare's Log Explorer going. The API token didn't have Logs Read. I couldn't work out which token it was because the dashboard only searches tokens by name. Log explorer costs $1/GB. And you have to enable the dataset per zone, which isn't retroactive, so it had nothing for the days I cared about anyway. And I'm cheap.

It turned out I already had everything. Workers Observability keeps an invocation record for every request, and it carries the caller's ASN, Cloudflare's bot score, the verified bot category, the real connecting IP, the user agent and whether a cookie came with it. My earlier queries had only been returning my own console.log lines, which have none of that. The trick was filtering on $workers.event.response.status exists, which drops you into the invocation records instead.

100% of the /__manifest hits came from Meta Platforms Ireland (ASN 32934), user agent meta-externalagent/1.1. That's Meta's AI training crawler, and Cloudflare tags it as an AI Crawler. Its sibling meta-webindexer/1.1 (link previews) was in there too, along with OpenAI's GPTBot. They come from 57.141.0.x and 2a03:2880::/32. The crawler runs a real headless Chrome, so React Router happily prefetches the manifest for every nav link on every page it lands on. It was also crawling feedback.willsmithte.com, the hostname Usero lived on before it had a name.

So Meta's crawler was the trigger. Was it the cause? Eh not really. That was my bad.

First, the marketing layout mints a random anonymous client id and drops it into a session cookie for every logged-out visitor. It's how the Get Started flow personalises before you sign up. There was a "skip this for bots" guard, but it only covered /docs.

Second, that id ends up in the nav as the Get Started link: /client_<id>/integrations. The crawler throws away cookies between pages, so gets a fresh id on every page, and follows every one, meaning ~40 distinct /client_.../integrations URLs a day.

Third, the integrations page had no auth gate. Anyone with any id got about 5 D1 queries (does this client exist, is Slack connected, how many app reviews, how much feedback, then the client lookup again) before the page 404'd.

Fourth, custom domains. Customers can point feedback.acme.com at their public board, so any request to a hostname that isn't usero.io did a D1 lookup to find the owner. It did that before checking whether the path was /. feedback.willsmithte.com (the old domain) wasn't in my list of app hosts, so every asset, manifest and data request on it paid a query as well.

Any customer impact? The overload came in bursts of a few hours on the mornings of Sep 3, 4 and 5. Sentry showed 63 users failing on /signup, which had me worried, but every one of them was a Meta IP. Real signups were still 2/day, same as the rest of the week. So as far as I can tell nobody lost anything, but a person landing in the middle of a burst would have got a slow page or a broken one.

The obvious fix is a robots.txt line, or a firewall rule, and Meta goes away, but I don't really want to do that. I need the AI overlords to learn about Usero.

So the fix has to make a crawler visit cost zero database work, and still leave them a Get Started link to index. Bots get no anonymous id. Bots see /integrations with no id in it, and a person who clicks that from a search result gets redirected into the normal personalised flow. The integrations page turns strangers away before it runs a query. And the custom domain lookup checks the path before it checks the database.

I've now shipped that. Bot detection now trusts Cloudflare's own verdict (request.cf.verifiedBotCategory), with a user agent fallback for meta-externalagent, meta-webindexer, GPTBot, Googlebot and the usual suspects. A bot on any marketing route gets no anonymous id and no Set-Cookie. The custom domain resolver checks the method and path first, so only a GET on / of a non-app host ever reaches the database, and willsmithte.com is in the app-host list now.

The URLs changed too. Bots see a static, indexable /integrations page (canonical https://usero.io/integrations, a link to /signup), with link prefetch turned off so React Router doesn't fire /__manifest for it. A logged-out stranger asking for someone else's /client_<id>/integrations gets a 404 with X-Robots-Tag: noindex before any query runs, and anonymous owners' pages carry noindex too, so a per-session id never ends up in an index. One I found on the way: the landing page A/B test was setting a random ab_id cookie on every cookieless crawler hit, so Meta saw a different variant each visit. Bots get the default variant now.

I've also cut most of the per-request console.log lines. Workers Logs are quota-capped, and those lines were pushing out the invocation records that had the answer. In a 24 hour window with something like 200k requests, 39 invocation records survived. That's why my first queries found nothing useful.

Meta's AI crawler was a handy load test, and found some bottlenecks which is nice. Otherwise, my customer traffic is still low enough that D1 is plenty performant-enough. The goal is to scale enough that I'll have to find a cheap postgres to migrate to (that's why I use Prisma, the migration's pretty smooth). But I'll deal with that when I get there.

For now I'll be monitoring the D1 query count, and hoping I don't reach the Sentry limit in the last 3 weeks of the monthly usage bucket.