A CDN migration can make your site faster while increasing the work your origin server has to do. The edge responds quickly, Google's crawlers may gradually increase their request rate, and an empty cache still sends uncached requests back to the same backend.
Google described this failure pattern in its November 11, 2026 CDN guidance. The useful response is to check cache behavior, origin capacity and verified crawler access together before the switch.
What Google explained about CDN migrations
Google's post connects two things that teams often monitor separately: how quickly the crawler receives a response and how much work the backend performs to produce it. A fast edge can improve the first measurement before it reliably reduces the second.
Google's crawl budget documentation distinguishes crawl capacity, the amount of crawling a host can support, from crawl demand, how much Google wants to crawl. Stable, fast responses can allow capacity to rise. Demand still matters, and the adjustment takes time. Moving behind a CDN does not guarantee a crawl spike or an increase in indexing.
This is especially relevant to large catalogs, frequently changing sites and launches with an already busy origin. A small blog that is crawled promptly usually has little reason to chase a larger crawl budget. Its CDN rollout still needs to preserve access to pages, images and other resources.
Why the cold-cache phase can overload the origin
A cache hit lets the edge serve a stored response. A cache miss requires it to obtain a response from the origin before it can serve and potentially store that response. At launch, many URLs have no cached copy. Crawlers exploring a broad URL inventory can therefore reach the backend much more often than visitors concentrated on a few popular pages.
The important qualification is cacheable. A CDN can cache images and scripts while forwarding every HTML request. Cookies, cache-control headers, cache keys and explicit bypass rules can also prevent reuse. A warm image cache does not establish that product pages or articles are being served from cache.
Cache state also varies between locations and cache layers. Fetching one URL successfully from your laptop does not prove that it is warm everywhere. A subsequent purge, expiry or changed cache key can create another period of misses.
My practical reading: treat the migration as a capacity change with a transition period. The CDN's average response time alone cannot tell you whether your application workers, database or upstream services are safe.
A simple model for origin load
For a first planning estimate, separate requests eligible for caching from requests that always bypass it:
Origin fetches per second ≈
cache-eligible requests per second × cache-miss fraction
+ requests per second that bypass the cache
The following numbers are hypothetical inputs, not measurements from SEO Experiments. Assume 30 cache-eligible requests and five bypassed requests per second:
| Scenario | Miss fraction | Calculation | Origin fetches / second |
|---|---|---|---|
| Mostly cold cache | 80% | 30 × 0.80 + 5 | 29 |
| Mostly warm cache | 10% | 30 × 0.10 + 5 | 8 |
Both scenarios receive the same incoming request rate. Their backend load differs substantially. If crawler traffic grows during the cold phase, the origin has less spare capacity than an edge latency graph suggests.
This model counts fetches, not processing cost. An uncached search page may be much more expensive than a static image. Request coalescing, tiered caches, revalidation and stale responses can also change the relationship. Use the calculation to identify missing measurements, then replace its assumptions with your own logs and application metrics.
Keep verified crawlers out of browser challenges
A sudden rise in automated traffic makes a broad bot challenge tempting. Google's November post warns that its crawlers cannot solve CAPTCHA challenges. Persistent blocks can leave Google seeing an error or challenge page instead of the content it needs to crawl.
Configure any exemption for the verified crawlers you want to allow. A request containing Googlebot in its user-agent string is not proof of identity. Google's verification documentation describes checking published IP ranges or performing reverse DNS followed by forward DNS confirmation.
Behind a CDN, the origin may see the proxy's address. Use the CDN's verified-bot classification or correctly configured, trusted client-IP handling. Do not trust an arbitrary client-supplied forwarding header as evidence that a request came from Google.
Inspect the exact rule that matched and the response body it produced. A status check alone can miss a challenge served with an otherwise successful response. A local request with a spoofed user agent also cannot demonstrate that a managed verified-bot exemption works for real Google traffic.
A rollout checklist that joins SEO and operations
This is the sequence I would use to turn Google's guidance into a migration plan:
- Capture a baseline. Record origin request rate, application latency, worker saturation, errors and current crawler activity. Include ordinary traffic peaks and a representative mix of page types.
- Check what will actually cache. Test public HTML, images, CSS and JavaScript separately. Review headers, cookies, cache keys, TTLs and bypass reasons. Keep account, cart and personalized responses out of shared caches.
- Review access rules before cutover. Check crawler verification, WAF exceptions and access to the public pages and resources that Google needs. Keep a record of the intended rules so a later emergency toggle can be traced.
- Warm a bounded set of useful URLs. Google recommends controlled pre-warming. Prioritize popular pages, frequently crawled URLs and upcoming campaign pages. Use real cache keys and a rate the origin can support. A cache-busting query string defeats the purpose.
- Keep the migration easy to diagnose. Where practical, separate the CDN switch from a redesign, URL migration or canonical change. Compare representative HTML, redirects, canonical tags, robots directives and sitemap responses before and after.
- Define rollback conditions. Agree on thresholds for errors, backend saturation and unexpected crawler blocks. Keep the previous configuration and a practical rollback procedure available.
If URL changes are part of the project, review their mappings separately. The Redirect Tool can help prepare that work, but it does not measure cache behavior or validate firewall rules.
How to diagnose the first hours after launch
Compare the same time windows at the edge and the origin. Group requests by page type, verified crawler and cache outcome before interpreting a site-wide average.
| Observed pattern | First check | Useful response |
|---|---|---|
| Fast edge responses; origin traffic and latency rise | Misses and bypasses for HTML, plus expensive URL patterns | Correct unintended cache bypasses and pace warming against measured capacity. |
| Static assets hit cache; HTML repeatedly bypasses it | Cookies, response headers and page-level rules | Confirm whether HTML caching is intended and safe before changing it. |
| Verified crawlers receive challenges or 403 responses | The matching WAF or bot rule and its exception | Restore access for the intended verified crawler; verify the saved rule and subsequent requests. |
| Successful responses contain stale or incorrect content | Cache keys, variants, invalidation and origin output | Fix the response selection; a 200 status does not establish content correctness. |
| Search traffic changes while access stays healthy | Queries, demand, deployments and indexing evidence | Investigate the change without assigning a ranking cause from timing alone. |
A technical crawl can check linked URLs, canonicals, directives and content consistency. It should be paced to suit the origin's condition. Our Screaming Frog audit workflow covers the crawl side; CDN logs and application monitoring supply the infrastructure evidence it cannot provide.
When a temporary crawl brake is necessary
If genuine crawler traffic is contributing to critical overload, Google's crawl-rate reduction guide documents temporary error responses. The CDN post specifically recommends 429 Too Many Requests or 503 Service Unavailable for a short emergency window: hours, or one to two days at most.
The documentation also describes Retry-After for 429 and 503 responses. For example:
HTTP/1.1 503 Service Unavailable
Retry-After: 120
That header indicates when to retry; it does not set a precise requests-per-second schedule. Google says a significant number of these errors can reduce crawling across the whole hostname, including URLs that still return content.
Make the brake temporary and observable. Assign an owner, expiry and recovery check. Confirm normal responses return when the origin recovers. Persistent errors are an availability problem, and an edge-cached error can outlast the overload that triggered it.
Review how emergency responses are cached before applying the rule. Then repair the underlying capacity, URL-inventory or cache problem. Keeping Google slowed indefinitely is not a successful CDN rollout.
Measure a successful migration at three layers
At the edge: track request rate, hits, misses, bypasses, challenge actions and status codes. Break out HTML from assets and check relevant locations or cache layers.
At the origin: track incoming fetches, application response times, errors, worker queues and expensive endpoints. The outcome you want is sustainable backend work while visitors and crawlers receive the correct content.
In Search: review Search Console's crawl and indexing evidence alongside logs. Inspect important URLs if access or content changed. Clicks and impressions are later outcome measures affected by many other factors; they do not validate cache configuration on launch day.
For a small controlled check, select a handful of public URLs across key templates. Observe the cache outcome on a first and repeated request, compare the response content, and match the requests to origin logs. Repeat from another relevant location if your CDN exposes location-level evidence. This tests specific paths and cache behavior; it is not a load test or proof of global warmth.
The decision is straightforward: call the rollout healthy when correct content remains accessible, backend load stays within capacity and cache behavior matches the intended rules. More crawling is useful only when the site can serve it reliably.
Sources and scope
Published November 21, 2026. This article analyzes Google's guidance and proposes an operational checklist. The numeric example is illustrative; no production CDN migration or ranking experiment is claimed.
- Google Search Central: Moving to a CDN without overwhelming your origin (or blocking Googlebot) — November 11, 2026.
- Google Crawling Infrastructure: Optimize your crawl budget — definitions of crawl capacity and demand.
- Google Crawling Infrastructure: Reduce the Google crawl rate — temporary response codes, Retry-After and hostname-wide effects.
- Google Crawling Infrastructure: Verify requests from Google crawlers and fetchers — DNS verification and published IP ranges.