Build with AI · Guide for product managers

AI fundamentals for product managers

Almost every product team is adding AI features. The PMs who do it well are not the ones who know the most model names — they understand how language models behave, what they cost, where they fail, and how to measure quality. This guide covers those fundamentals.

Guide 8 of 13 in the technical PM path

12 min read · Updated 25 September 2026 · By the TechPMer team

What a large language model actually does

A large language model (LLM) — like the models behind ChatGPT, Claude or Gemini — is trained on huge amounts of text to predict the next piece of text. Given an input, it generates the most plausible continuation, one small piece at a time.

That simple mechanism produces surprisingly capable behaviour: summarising, drafting, classifying, extracting information, writing code, reasoning through steps. It also explains the main weakness: the model produces plausible text, not verified facts.

Tokens and the context window

Models read and write tokens — chunks of text, roughly ¾ of an English word on average. Tokens matter to PMs for three reasons:

• Cost: providers charge per token, for both input and output.
• Speed: longer outputs take longer to generate.
• Context window: each model has a maximum number of tokens it can consider at once — your instructions, the conversation so far, any documents you include, and its answer. Anything outside the window, the model simply does not see.

"Why did it forget what I said earlier?" is often a context window question: the conversation got too long, or the app did not send the history.

Prompts and system instructions

A prompt is everything sent to the model. Products usually combine a hidden system prompt (role, rules, tone, format), relevant context (the user's data, documents), and the user's message. Much of an AI feature's quality comes from this assembly: clear instructions, good examples, the right context, and an explicit output format.

Hallucinations

A hallucination is a confident but false answer — an invented fact, citation, API or number. It is not a bug that will be fixed in the next release; it is inherent to how the models work. Product design has to account for it:

• ground answers in your own data (see RAG below) and show sources;
• ask the model to say when it does not know;
• keep a human in the loop for high-stakes actions;
• make it easy for users to correct or report answers.

RAG: answering from your own data

Retrieval-augmented generation (RAG) means: before asking the model, search your own content (help articles, documents, product data) for the most relevant pieces, and include them in the prompt with an instruction to answer from them.

RAG is how most "chat with our docs" and support assistants work. Its quality depends less on the model and more on the retrieval: whether the right passages are found. Common PM questions: what content is included, how fresh it is, and what happens when nothing relevant is found.

Structured output, tools and agents

• Structured output: asking the model to reply in a fixed format such as JSON (for example { "sentiment": "negative", "topic": "billing" }) so your software can use the result reliably. Always validate it.
• Tools (function calling): the model can request actions your code performs — search orders, create a ticket, check the weather — and then use the results.
• Agents: a model working in a loop: plan, call tools, look at results, repeat until the task is done. Coding agents such as Claude Code and Codex are agents. More autonomy means more capability and more ways to go wrong, so permissions, limits and review matter.

Cost, latency and limits

AI features have running costs that traditional features do not:

• Estimate cost per use (input + output tokens × price) and multiply by expected usage.
• Protect against abuse with rate limits and per-plan quotas.
• Consider smaller, cheaper models for simple tasks and larger ones only where quality demands it.
• Design for latency: stream responses, show progress, move long tasks to the background.

Evaluation: how you know it works

You cannot test AI features only by clicking through them. Build an evaluation set: 30–200 realistic inputs with what a good answer looks like. Run it whenever you change the prompt, model or retrieval, and compare results. Combine automated checks (format, required facts, forbidden content) with human review of a sample. Track real-world signals too: thumbs up/down, edits, retries and support tickets.

Privacy and safety

• Know what user data you send to the model provider and whether it may be used for training (API terms usually differ from consumer apps).
• Remove secrets and unnecessary personal data before sending.
• Treat model output as untrusted: validate it, and never let it run code or take destructive actions without checks.
• Watch for prompt injection: content (a web page, an email, a document) containing instructions that try to hijack the model. Keep data and instructions separate and limit what tools the model can use.

What engineers may tell you

• "We're hitting the context limit." Too much text for one request; summarise, retrieve less or split the task.
• "The retrieval isn't finding the right chunks." A search problem, not a model problem.
• "Let's use structured outputs with a schema." More reliable parsing and fewer broken responses.
• "We need evals before switching models." Measure quality instead of guessing.

Questions a good technical PM asks

1. What exactly is the user problem, and why is AI the right tool for it?
2. What happens when the answer is wrong — how does the user notice and recover?
3. What data goes to the model, and what are the provider's data terms?
4. What does one use cost, and what are our limits per user and plan?
5. How will we measure quality before and after launch?

Red flags

• "Let's add AI" with no specific user problem.
• No plan for wrong answers.
• No evaluation set — quality judged by a few demos.
• Unlimited free AI usage with no rate limits.
• Model output inserted into pages or used to take actions without validation.

Try it yourself

Pick one repetitive task in your product (tagging feedback, drafting replies, summarising notes). Write down 20 real examples and the ideal output for each. Try a prompt in any AI assistant against all 20. You have just done a small evaluation — the most important habit in building AI products.

In the TechPMer course, week 10 (“Add an AI Feature”) adds an AI feature to your own project with prompts, limits and an evaluation checklist. See also Claude Code and Codex for product managers.