A business wanting its own content to answer customer questions is usually told it needs to train a model. It almost never does. It needs the content in a structured, attributed form that an assistant can read at the moment a question is asked.
That distinction decides the architecture, the cost, the update cycle and whether an answer can cite where it came from. It is worth being precise about before anyone builds anything.
The distinction
Reference material is not training data.
Retrieval keeps your content as documents. When a question arrives, the relevant passages are found and given to the model along with the question, and the answer is generated from them. The content stays yours, stays readable, and can be pointed at as the source of an answer.
Training, or fine-tuning, folds examples into a model's weights. It changes how the model behaves — its tone, its format, its handling of a task — and it is genuinely useful for that. It is a poor way to store facts: you cannot cite a weight, you cannot correct one page, and updating anything means training again.
Nearly every "we want AI to answer questions about our business" requirement is a retrieval requirement. Getting this wrong is expensive in the specific way that is hard to reverse.
| Retrieval / reference | Training / fine-tuning | |
|---|---|---|
| What it changes | What the model is shown at question time | The model's weights |
| Updating a fact | Edit the document | Retrain |
| Citing a source | Natural — the passage is right there | Not possible |
| Removing content | Delete the document | Retrain, and hope |
| Good for | Facts, prices, policies, procedures | Tone, format, task behaviour |
| Typical cost | Storage and a search index | Compute, expertise and repetition |
Step 01
Extract the pages that answer questions, not the pages that market.
The pages that make a good knowledge base are rarely the ones that get the most traffic. FAQ, support, policy, pricing, process, contact and service-detail pages carry the answers. Homepages and campaign landing pages carry positioning, which produces confident-sounding answers built out of adjectives.
A bounded selection also beats a full crawl. Five well-chosen pages of genuine reference content outperform two hundred pages where the useful passages are diluted by navigation, testimonials and calls to action.
- FAQ and support pages, which are already written as question and answer.
- Policy pages: returns, delivery, cancellation, warranty, privacy.
- Service or product detail pages with real specifics, not benefit statements.
- Process and onboarding pages describing what actually happens.
- Contact and location pages, including hours and coverage.
- Skip: campaign pages, blog archives, anything already outdated on the site itself.
Step 02
Structure beats volume, and attribution beats both.
A knowledge base is retrieved in pieces. A page dumped in as one long block competes with itself: the passage that answers the question sits alongside four that do not, and the retrieval step has to guess.
Split on the document's own headings, keep lists and tables as lists and tables, and keep existing FAQ pairs paired — they are already in the shape a retrieval system wants.
Every section should carry the URL it came from. Without it you cannot verify an answer, cannot update the right page when something changes, and cannot show a customer where a claim originated. Attribution is not a nicety; it is what makes the system maintainable.
Step 03
What extraction cannot reach.
Content built in the browser by JavaScript is invisible to a fetcher that does not run one. If a page's main content only appears after a script runs, the extractor sees the shell. This is common on single-page applications and on sites whose content loads from an API after render.
Access-controlled content stays out of reach, and should. Logins, paywalls and member areas are not bypassed, and robots rules are followed whether or not you own the site.
Anything behind a form, in a PDF, in an image, or generated per-visitor also falls outside a public-HTML extraction. Those need a different route in — usually an export from the system that holds them.
Progressive disclosureTechnical Notes
Bounded by design
The AI Knowledge Studio reads the site's robots.txt and follows it, discovers up to fifty candidate URLs using two discovery requests, and fetches only the pages you select — currently up to five per run, a couple at a time.
The permission box is a statement, not a check
Confirming you are authorised records your statement. It does not verify ownership and it relaxes nothing: robots rules apply either way.
Stateless extraction
No job record, no raw HTML and no copy of the extracted content is kept on the server. The knowledge base exists in your browser tab until you export it.
Step 04
The part everyone skips: keeping it true.
A knowledge base is a copy. The moment a price, a policy or an opening time changes on the website, the copy is wrong — and it will keep answering confidently, because nothing about being out of date makes an answer look uncertain.
Decide the refresh cycle before launch, and make it someone's job. Re-extract the pages that change, not the whole site. Keep the source URL on every section so the diff is obvious.
It is also worth deciding what the assistant should do when it does not know. An honest "I don't have that — here is how to ask" is better for a customer than a fluent answer assembled from adjacent content, and it is a configuration decision, not a model limitation.
- A named owner and a refresh cadence, agreed before it goes live.
- Source URLs retained on every section so updates are traceable.
- A defined fallback for questions outside the knowledge base.
- A review of the answers people actually ask for, not the ones you expected.
Keep reading