Ranking a website in ChatGPT, honestly: a site level triage of what a crawler actually receives
Before this is a content question it is a site question. Triages your URLs into cannot be fetched, fetched but empty, and readable but not quotable.
- Works in
- ChatGPT, Claude, Gemini
- You need
- Your robots.txt, pasted · A raw HTML word count per URL from curl · The URLs you most want quoted · Your framework, if the site is client rendered
- Written for
- how to rank website on chatgpt
Scored by our own engine
This page, run through the audit we sell. Measured 4 August 2026.
Before any of this is a content question it is a site question. A crawler that is disallowed never arrives, and a crawler that arrives at a client rendered page receives the empty shell your server sent. Neither failure is visible in a browser and neither is fixed by writing better paragraphs. The prompt below triages your URLs into what cannot be fetched, what is fetched but empty, and what is readable but not quotable, and it refuses to judge any page you have not actually measured.
The two failures that happen before content matters
Permission is binary and it is documented. OpenAI, Anthropic, Perplexity and Google all publish which crawlers they run and how robots.txt controls them, so whether GPTBot or OAI-SearchBot may fetch your page is a fact you can check in two minutes. The trap is the matching rule: a crawler obeys exactly one group, chosen by the longest matching token, and ignores every other group including the wildcard. That is how a file meant to welcome one bot ends up blocking everything else.
The response is the second failure and it is undocumented rather than settled. None of the three assistant operators has published a sentence about whether their crawlers execute JavaScript. Google has, in detail, which is why a site can rank acceptably and still return almost nothing to a plain fetch. We measured one well known profile page returning 5 words before JavaScript ran and 989 after.
You cannot build on an undocumented behaviour. Designing for a plain fetch is a decision taken under that uncertainty, not a claim about what OpenAI does.
Why the prompt refuses to look at your site
Ask a model to check a URL and it will produce an assessment. Sometimes it fetched something, sometimes it fetched a rendered version that differs from what a crawler receives, and sometimes it described a page it never saw. All three read identically in the output.
So the prompt is given the measurement instead of the URL, and any page without one goes into an UNMEASURED bucket whose only content is the command to run. That is a worse looking report and a true one. The number takes about ten seconds per page to produce with curl, and it is the same number our render gap checker reports alongside the hydration markers that separate a thin page from a full page whose content has not arrived.
Work the buckets in order. When access and rendering are clean, the remaining question is whether ChatGPT is retrieving at all or answering from recall, which is a different mechanism, and the overview page puts the rest of it in context.
You are a technical SEO engineer triaging a site for whether an AI assistant
can read it at all. This is a site level and infrastructure job, not a writing
job. Do not give me content advice anywhere in this output.
You cannot fetch my pages and you must not pretend to. Judge only the evidence
I paste. If I have not given you a measurement for a URL, put it in the
UNMEASURED bucket and stop there.
Sort every URL I give you into exactly one of four buckets, and return them as
four blocks in this order. No table.
BUCKET ONE, CANNOT BE FETCHED. URLs excluded by my robots.txt for any of
GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot. Apply the real matching
rules: a crawler obeys exactly one group, chosen by the longest matching user
agent token, and it ignores every other group including the wildcard. Within
the applying group, the longest matching path wins rather than the first line
in the file. For each URL name which crawlers are shut out and quote the line
that does it.
BUCKET TWO, FETCHED BUT EMPTY. URLs where the raw HTML word count I gave you
is under 250. Say what the number is, what the page is supposed to contain,
and which of the four fixes fits my stack: server side rendering, static
generation at build time, prerendering for crawlers, or moving the important
text into the first response by hand. Give the cost of each in engineering
terms, not in adjectives.
BUCKET THREE, READABLE BUT NOT QUOTABLE. URLs with a healthy raw word count
where the structural problem is what remains. Limit yourself to structure I
can verify from the outside: heading levels, whether headings ask anything,
whether an author is named, whether there is structured data. Do not rewrite
my copy.
BUCKET FOUR, UNMEASURED. Every URL I did not give you a word count for. The
only line under each is the command to run.
Then one closing section, THE ORDER. The buckets in the order I should work
them and why, in under 100 words. Bucket one before bucket two before bucket
three, always, because permission gates the fetch and the fetch gates
everything else.
Constraints:
- Do not tell me any of this will make ChatGPT rank, cite or recommend my
site. No operator publishes a ranking system. Frame every fix as removing an
obstacle, which is what it is.
- Do not state that any assistant crawler does or does not execute JavaScript.
OpenAI, Anthropic and Perplexity publish nothing either way. Say it is
undocumented and explain that designing for a plain fetch is a decision made
under that uncertainty.
- Do not invent statistics, adoption figures or expected uplift.
- Do not guess at the contents of any URL. You have not seen them.
My robots.txt: [PASTE THE ENTIRE FILE, OR "none served"]
My URLs and raw word counts: [ONE PER LINE: URL, THE CURL WORD COUNT, AND WHAT THE PAGE IS FOR]
My stack: [FRAMEWORK AND HOSTING, OR "not sure"]What to change
Everything in square brackets is yours to replace. Nothing else needs editing.
[PASTE THE ENTIRE FILE, OR "none served"]- The whole file, including groups that look irrelevant. Group precedence is the trap: a per bot group means the wildcard rules stop applying to that bot entirely, so a file that blocks GPTBot from everything can also hand it the private directory the wildcard was protecting. A partial paste hides exactly that.
[ONE PER LINE: URL, THE CURL WORD COUNT, AND WHAT THE PAGE IS FOR]- The measurement this whole prompt runs on. Read the number against what the page is for: 1,400 words on a homepage is fine, 40 on a long article is not, and 12 on a product page is the finding. Without the "what the page is for" part the model has no basis to call any number low.
[FRAMEWORK AND HOSTING, OR "not sure"]- Decides which of the four rendering fixes is a migration and which is a rewrite. On Next.js or Nuxt, server side rendering is a configuration and routing change. On a Vite plus React single page app it is closer to rebuilding the application, and the honest recommendation is usually different as a result.
How to run it
- 01Fetch your robots.txt and copy the whole thing
Open yourdomain.com/robots.txt and copy every line, including the groups you think are irrelevant. If nothing is served, write "none served", which means everything is allowed by default and bucket one will be empty. That is a legitimate and common state rather than a problem.
- 02Measure the raw word count for each URL
Run a plain curl against each page and count the words of visible text in the response, before any JavaScript executes. Do not open the page in a browser and estimate: the browser shows you the rendered version, which is the version some readers never get, and that mistake is the entire reason this triage exists.
- 03Say what each page is for, in three words
A number on its own cannot be judged. "Homepage" and "4,000 word buying guide" make the same word count mean opposite things. This is also what stops the model calling a deliberately short contact page a failure.
- 04Run it and work bucket one first, even though it is boring
A Disallow line is one commit and it takes effect on the crawler's next visit. Everything in buckets two and three is engineering work that buys nothing while bucket one is non empty. This ordering is the single most common thing sites get wrong here.
- 05Price the rendering fix before you promise it
Bucket two is where the real budget lives. Server side rendering, static generation, prerendering and hand placing the important text into the first response are four genuinely different commitments. Take the model's framing to whoever owns the codebase and let them cost it rather than picking one from the output.
- 06Re measure rather than trusting the fix
After each change, run the same curl and compare the number. Run the render gap checker too, since it separates a genuinely thin page from a full page whose content has not arrived by looking for hydration markers in the source. Those are different problems with the same word count.
Questions people ask
How do I rank my website on ChatGPT?
There is no ranking to enter: OpenAI publishes no ranking system, no ranking factors and no submission form. At the site level the reachable version of the question is whether a crawler is permitted to fetch your pages, and whether those pages contain real text in the response before JavaScript runs. Both are measurable today and both are obstacles rather than levers.
Why would my site be invisible to ChatGPT if it ranks fine on Google?
Because Google documents that it renders JavaScript with headless Chromium from a queue, and OpenAI, Anthropic and Perplexity document nothing either way about rendering. A client rendered site can therefore hold decent Google positions while returning an almost empty document to anything that does not run a browser. It is the most confusing failure mode in this category because every person who checks the page sees it working.
How many words should be in my raw HTML?
Judge it against what the page is for rather than a universal threshold. As a working rule, under 250 words of visible text in the raw response on a page that should carry real content is worth investigating and under 80 is a failure. A homepage returning 1,400 words is healthy. A 4,000 word guide returning 40 is a rendering problem, not a content one.
Does adding llms.txt help if my pages come back empty?
No. A file pointing at pages that return nothing is decoration on a locked door. llms.txt is a proposed convention, no assistant guarantees it reads the file, and it belongs last in the sequence rather than first. Fix access and rendering, then publish it because it costs an hour.
Is prerendering an acceptable fix?
It works and it does not touch your application code, which is why it stays on the list. Google calls dynamic rendering a workaround rather than a long term solution and recommends server side rendering, static rendering or hydration instead. You also take on a second serving path that can drift from the real one, and the copy you never look at is the one the crawler reads.
Unsubscribe in one click. We never pass your address on.