Why AI crawlers see a blank page on your React site

If your content only appears after React, Vue or Angular hydrates, then anything that reads your page without running a browser gets the empty shell your server sent. That is not a theory. We measured it on live sites this morning and one well known profile page carried 5 words of readable text before JavaScript ran and 989 after.

What a crawler actually receives

A plain fetch returns only the HTML your server wrote. Everything assembled in the browser is absent from it. Here is the same set of pages measured twice, once with curl and once in a real headless Chromium.

Measured 4 August 2026. The curl column strips script, style and noscript blocks, removes the remaining tags and counts words. The rendered column is document.body.innerText after the network went idle.

Page	Words in raw HTML	Words after rendering
mastodon.social/@Gargron	5	989
open.spotify.com	4	640
soundcloud.com/discover	51	346
excalidraw.com	3	77
en.wikipedia.org/wiki/JavaScript	11,167	8,552
stripe.com	1,895	not measured
visibility100x.com	1,420	not measured

Two honest notes about that table. The two columns are not quite the same measurement, which is why Wikipedia’s raw count is higher than its rendered one: the raw HTML includes navigation and footer text that innerText skips because CSS hides it. The signal worth reading is the order of magnitude, not the ratio. And the bottom three rows are there as controls. They are server rendered or statically generated, so the text is already in the response and the browser adds nothing a crawler needed.

The Mastodon row is the one worth sitting with, because it is not an app with nothing to say. It is a profile with years of posts on it. The server sent 56 KB of HTML containing 5 words of readable text. Everything a reader would quote arrived afterwards.

This is also not limited to the render gap. An Instagram profile we fetched the same morning returned 393 KB of HTML containing exactly 1 word of visible text, and a headless browser hit a login interstitial rather than the profile, so we have no comparable rendered figure for it. Some pages are unreadable to a crawler for two separate reasons at once.

Run this on your own site

One line, no install beyond curl and python3, and it counts the words of visible text in what a crawler receives. Change the URL and run it.

curl -sL --compressed -A "GPTBot" https://example.com | python3 -c "import sys,re;h=re.sub(r'(?is)<(script|style|noscript)[^>]*>.*?</\1>',' ',sys.stdin.read());print(len(re.sub(r'(?s)<[^>]+>',' ',h).split()),'words')"

Real output from two runs of exactly that line, with only the URL changed:

$ curl ... https://mastodon.social/@Gargron | python3 -c ...
5 words

$ curl ... https://visibility100x.com/ | python3 -c ...
1420 words

Read the number against what the page is for. A homepage with 1,400 words is fine. A long article that returns 40 is not, and neither is a product page that returns 12. Under 250 words on a page that should carry real content is where we start treating it as a finding, and under 80 is a failure.

Which crawlers render JavaScript, according to their own documentation

Three of the operators say nothing about it. Checked 4 August 2026 against each operator’s own published page, linked at the end.

Crawler	Operator	Documented rendering behaviour
GPTBot, OAI-SearchBot, ChatGPT-User	OpenAI	Not documented
ClaudeBot, Claude-User, Claude-SearchBot	Anthropic	Not documented
PerplexityBot, Perplexity-User	Perplexity	Not documented
Googlebot	Google	Renders, with headless Chromium, from a queue
Bingbot	Microsoft	“generally able to render JavaScript”, with stated limits
Applebot	Apple	“may render the content of your website within a browser”

Be precise about what that first block means. It does not say those crawlers never render. It says their operators have written pages explaining what the crawlers are for, how to identify them and how to block them, and have not written a sentence either way about JavaScript.

You cannot build on an undocumented behaviour. If a capability is not published it can change without an announcement, and you find out from a traffic graph. So the position that follows is not a guess about what OpenAI does. It is a decision: design for a plain fetch, because a plain fetch is the only behaviour any of those three has committed to.

Google is the clear case and it is documented in detail. Google Search “runs JavaScript with an evergreen version of Chromium”, and “Googlebot queues all pages with a 200 HTTP status code for rendering”. The queue is the part people skim past. Google’s own wording is that a page “may stay on this queue for a few seconds, but it can take longer than that”.

Why rendering is the first thing an operator drops

Rendering a page costs orders of magnitude more than fetching it, and at web scale that difference decides the architecture. We instrumented the headless browser to count what it took to turn each page into text.

The Mastodon profile above needed 277 HTTP responses, 182 of them JavaScript files, roughly 10 MB of transfer and 6.6 seconds to produce its 989 words. The Wikipedia article needed 49 responses and 543 KB, and its 11,167 words were already sitting in the first response before any of that happened.

That is the economics in two numbers. One page costs a socket and a parse. The other costs a browser process, a network of subresource fetches and seconds of wall clock, per page, across every page of every site. Microsoft has said as much in public, calling it “difficult for bingbot to process JavaScript at scale on every page of every website, while minimizing the number of HTTP requests at the same time”. An operator under cost pressure drops rendering first, and nothing published suggests the assistant crawlers chose differently.

Why your Google rankings will not warn you

Google renders, so a client rendered site can hold decent positions while being close to empty to every assistant. That combination is the single most confusing failure mode in this category, because every human who checks the page sees it working perfectly.

The check that misleads people is opening the site, or viewing it in any tool that runs a browser. Both show the rendered page, which is the version some of your readers never get. The only view that answers the question is the raw response, which is why the command above pipes curl straight into a word count rather than into anything clever.

The failure is also gradual, which makes it harder to catch. A site in this state does not drop off a cliff. It holds its rankings and quietly stops being mentioned, and the two trends are hard to connect because they show up in different reports.

The fix ladder, most expensive first

There is no settings toggle. Every option below is a change to how your application produces HTML, and the cheap ones buy less than the expensive ones. Google’s own guidance names the top of this ladder: it recommends “server-side rendering, static rendering, or hydration”.

Server side rendering. The server runs your components and returns finished HTML on every request. It is the most complete answer and the most expensive to adopt, because code written for the browser tends to assume window exists, data fetching moves, and you now run and pay for compute you previously pushed onto the client. On Next.js, Nuxt or Remix this is a migration rather than a rewrite. On a Vite plus React single page app it is closer to a rewrite.

Static generation. Render every page to HTML at build time and serve files. This produces the best result of anything on this list and is often the cheapest to run, with one hard constraint: it only works for content that is the same for everybody. Marketing sites, documentation, blogs, product catalogues and pricing pages all qualify, and those are usually exactly the pages you want quoted. This site is built this way, which is why its homepage returns 1,420 words to curl.

Prerendering. Run a headless browser yourself, cache the resulting HTML, and serve that to crawlers while people continue to get the app. It works, it does not touch your application code, and it is the reason it stays on this list. Be aware of what you are taking on. Google calls dynamic rendering “a workaround and not a recommended solution, because it creates additional complexities and resource requirements”, and describes it as “not a long-term solution”. You are also now maintaining two serving paths that can drift apart, and the one you never look at is the one the crawler reads.

Move the important text into the initial HTML by hand. For many sites the content that needs to be quotable is a fraction of the page: the H1, the opening answer, the specifications, the price, the FAQ. Putting that specific text into the server’s first response, while the rest of the application continues to hydrate, is the smallest change on this list that genuinely moves the number. It is also the most fiddly to keep true, because it is a second copy of the content.

What the partial mitigations actually buy

They move you from invisible to thin, which is worth having and is not a fix. A crawler that gets a heading and two sentences can at least tell what the page is about.

A noscript block can carry a summary, a heading and links. Meta description and Open Graph tags are in the first response by definition and are sometimes the only real sentence on a client rendered page, which is why they are worth writing properly rather than templating. JSON-LD structured data emitted server side gives a machine your name, price and author even when the body is empty. An RSS or JSON feed of your content is a plain document a crawler can read.

None of these put your article in front of the crawler. They put a description of your article in front of the crawler. If your competitor’s page returns 1,800 words and yours returns a 155 character meta description, the passage that gets quoted is not yours. Do them anyway, and do not file them as done.

One thing not on this list is llms.txt. A file pointing at pages that come back empty is decoration on a locked door, and the same is true of opening your robots.txt to every AI crawler. Permission and discovery are worth nothing until there is something in the response to read.

Check yours, then decide what it is worth fixing

Run the one line command above. It costs nothing, it needs no account, and it is the same measurement we make.

Our render gap checker adds the part the command cannot do. Alongside the word count it looks for the markers of a client rendered application in the source, such as an empty root element, a Next.js or Nuxt hydration payload, or an Angular version attribute. That separates a genuinely thin page from a full page whose content has not arrived yet. Those are different problems with different fixes and the same word count. The full audit puts the result next to the rest of what an assistant sees when it reads your site.

If the answer is that your stack cannot produce HTML without a rewrite, that is a real engineering decision with a budget attached, and it is the case where our team doing the work is the honest recommendation rather than a tool. We would rather tell you that than sell you a toggle that does not exist.

Sources

Every claim about a crawler above comes from that operator’s own documentation. Check these rather than us, because they change and we are a snapshot dated 4 August 2026.

• OpenAI, bots and user agents, which describes GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User and says nothing about JavaScript execution.
• Anthropic, does Anthropic crawl data from the web, which describes ClaudeBot, Claude-User and Claude-SearchBot and says nothing about JavaScript execution.
• Perplexity, bots, which describes PerplexityBot and Perplexity-User and says nothing about JavaScript execution.
• Google, JavaScript SEO basics, for the three phases, the render queue and the evergreen Chromium quotes.
• Google, dynamic rendering as a workaround, for the workaround wording and the recommendation of server side rendering, static rendering or hydration.
• Microsoft, bingbot series: JavaScript, dynamic rendering, and cloaking, for what bingbot renders and the cost of doing it at scale.
• Apple, about Applebot, for the rendering and robots rules section quoted above.

The measurements are our own, taken on 4 August 2026 with curl sending a GPTBot user agent string and with Playwright driving headless Chromium at a 1280 by 900 viewport, waiting for network idle plus a short settle. Re run them yourself. They will have moved, and the point is the method rather than the numbers.