Ask most people how AI reads their website and they'll describe something close to a person: a model opens the page, starts at the top, and works its way down. It is a reasonable guess. It is also wrong in nearly every detail that matters.
What actually happens looks less like reading and more like an industrial process. A page is fetched, taken apart, converted into a form no human would recognize, filed away in pieces, and then — later, in response to a question you will never see — partially reassembled into an answer. The distance between those two pictures is where most visibility is now won or lost.
It helps to hold both mental models side by side.
A model opens your page and reads it top to bottom, forming an impression of the whole — the way a person skims an article.
A pipeline fetches your page, splits it into fragments, stores a mathematical version of each, and later retrieves a few to assemble an answer — often beside fragments from other sites.
That pipeline has six stages worth understanding. None is complicated on its own. Together they explain why the old instincts about search no longer hold.
Crawling
Before anything can read your page, something has to fetch it. A crawler is a program that requests your URL, downloads the HTML, and follows the links it finds to discover more. It collects; it does not comprehend.
This first step is quietly decisive, because a crawler can only pass along what it can reach. Content behind a script that never runs, a login, or a link it cannot follow may as well not exist. Speed and structure matter here for an unglamorous reason: a crawler working through millions of pages will not wait around for yours.
A crawler can only consider what it can reach. Everything else is invisible — not ranked poorly, but absent.
Parsing
Once a page is fetched, it has to be parsed — turned from a tangle of markup into something with shape. The parser separates your words from the code around them and reads the structure you gave them: which text is a heading, which is a list, which is a caption, and what follows what.
Structure, at this point, stops being decoration. A heading is not just larger text; it is a claim about what the passage beneath it is for. Order implies relationship. The clearer your structure, the more accurately a machine can infer what each part of your page actually means.
Chunking
A page is rarely handled as a single unit. It is split into smaller passages — chunks — each a few sentences to a few paragraphs long. Part of the reason is practical: models work within limited windows of attention, and smaller pieces are easier to store and compare. But it carries a consequence most sites have not absorbed.
You are not retrieved as a page. You are retrieved as a passage. A single strong paragraph can be surfaced on its own, lifted clean away from everything around it — and a paragraph that only makes sense in the context of the whole page can be lifted away too, and fail.
Write so that each passage can stand on its own. The paragraph, not the page, is the unit that gets found. This is the heart of Content Engineering — content built to survive being taken apart.
The unit of visibility is no longer the page. It is the paragraph.
Embeddings
Here is the step that feels like magic and isn't. Each chunk is converted into an embedding — a long list of numbers that captures its meaning. Passages about similar ideas end up with similar numbers, which means they sit near one another in a vast mathematical space, whether or not they share any of the same words.
The practical effect is a shift you can feel in your writing. Someone searching for how to get cited by AI can be matched to a passage that never uses that phrase, because the meaning lines up. Exact keywords lose their grip; saying one thing clearly gains it.
An embedding is a coordinate for meaning. Two passages that say the same thing in different words land near each other. A page stuffed with keywords but thin on meaning lands nowhere useful.
Retrieval
Now the pieces come together. When someone asks a question, the system turns that question into an embedding too, and looks for the chunks whose embeddings sit nearest to it. It does not read your whole site. It pulls a small set of the most relevant passages — often a handful, frequently from several sources at once.
This is where traditional search and AI search visibly part ways. A search engine returns a list of links and lets the reader choose. Retrieval hands passages to a model, and the reader never sees the list. Your paragraph is no longer competing for a click. It is competing to be one of the few fragments a machine decides to use.
Generation
Finally, the model writes. It takes the retrieved passages and composes an answer in its own words — sometimes quoting, sometimes paraphrasing, sometimes naming its sources and sometimes not. It is reconstructing, not copying: assembling a response from fragments, the way someone might brief you from notes they took across a dozen articles.
Whether your passage makes it into that answer — and whether you are credited when it does — depends on everything upstream: that you were reachable, clearly structured, coherent in pieces, unambiguous in meaning, and trusted enough to be chosen. Earning that final citation is the work we call Answer Engine Optimization.
Being on the page was the old goal. Being in the answer is the new one.
So what should companies actually do differently?
None of this rewards tricks. If anything it punishes them — a system that takes your writing apart and reassembles it has little patience for pages built to game a ranking. A few things genuinely change:
- Write passages that stand alone. Assume any paragraph might be lifted out of context and still need to make sense.
- Make structure real. Headings, order, and lists are read as meaning, not styling, so they should reflect how the ideas actually relate.
- Say one thing clearly. Meaning, not keyword density, is what gets matched. Ambiguity is expensive.
- Earn trust off the page. Retrieval favors sources the rest of the web already treats as credible.
- Measure the answer, not just the ranking. Where you appear in AI responses is now its own metric.
These are not five separate projects. They are one posture — building a company's expertise so it survives the trip through a machine and comes back out intact. Deciding where that effort is worth spending, and in what order, is usually the first conversation we have; it is the substance of Strategic Consulting. The rest of this publication is us working out the details in the open.