On this page
Part 1 of a 4-part series on agentic programming.
It's been said that a prompt is not a wish. It's an interface — a defined input, a defined output, and some way to tell whether a change to it made things better or worse. Most of what separates an LLM feature that works from a shiny demonstration comes down to taking that seriously.
The wish version is the natural place to start, and not a naive one. It's what a careful person does the first time: describe the task in good English, read what comes back, find it reasonable, and move on. Nothing in that exchange announces what's been left out. That surfaces later, when the same prompt has to run four thousand times and nobody reads any of them. It's caused by success rather than by anyone being careless.
What surfaces is usually two failures, and they're worth separating, because the fixes are different. The first is that the output has no structure. It's prose, and prose can't be sorted, routed, stored in a column, or compared against the previous version of itself.
The second is that the task has no definition. The model has been using its own idea of what a good answer looks like, well enough that nobody noticed — right up until you disagree with it about a case that matters.
The second one has the longer tail, because its fix isn't "write a better prompt." Its fix is being able to find out for certain. Most of us are familiar with the usual advice about prompting: be specific, be clear, give the model a persona. The advice isn't wrong, but none of it is checkable — two prompts can both be specific and clear, and one of them can be worse. A great place to be in engineering is the point where you can be wrong on purpose and find out cheaply.
That's what we're building in this part. By the end we'll have a function that takes a messy inbound sales lead and returns a typed qualification with readable reasoning, a small set of examples labeled by someone whose judgment we trust, and a number that tells us whether the last change to the prompt helped. Then we'll change something, watch the number get worse, and revert it. That loop is the argument.
Everything is TypeScript calling the Claude API directly via messages.create — no agent framework and no SDK agent loop. Across the four parts we build the loop by hand. It's shorter than you'd expect, and seeing it once removes most of the mystery from everything built on top of it.
A good problem to have
We are going to build one program across the four parts of this series: an automated sales lead tool. It starts here, as a single API call that scores a lead. By Part 4 it's a supervised set of agents that researches companies, qualifies them, drafts outreach, and asks a human before it sends anything.
The fictional company we're doing this for sells a data-pipeline observability product. The specifics are relevant, and they don't change for the rest of the series:
- We sell to data platform teams — the people who own Airflow, dbt, Spark, and the warehouse.
- Our customers have roughly 200 or more engineers. Below that, teams don't have a data platform team, they have one overworked person.
- Average contract value is about $60,000 a year.
- The buyer is a Director of Data Platform, a Head of Data Engineering, or a VP with those words nearby.
Inbound leads arrive from a form on the website. The form asks for a name, an email, a company, and has a free-text box labeled "how can we help?" What actually comes out of that box is anything from a two-word question to a three-paragraph incident report. Some of these leads are worth a salesperson's afternoon and some are a student looking for a free tier, and the only way to tell is to read them.
The example is particularly relevant for reasons that only show up later. Leads are genuinely messy unstructured input. Qualification is based on judgment, not rules. Enrichment needs real tools. And the last step — sending an email to a real person at a real company — can't be undone. That last one is why the guardrails in Part 3 are so important.
For now, a lead is this:
// src/lead.ts
export interface Lead {
id: string;
name: string;
email: string;
company: string;
title?: string;
notes: string;
}
export const SAMPLE: Lead = {
id: "L-1041",
name: "Priya Raman",
email: "priya.raman@northwindlogistics.com",
company: "Northwind Logistics",
title: "Director of Data Platform",
notes:
"We run about 40 Airflow DAGs feeding dbt into Snowflake. Right now the way we " +
"find out a pipeline broke is that finance tells us their dashboard is wrong. " +
"I have budget approved for this quarter and I'm looking at two other vendors.",
};
That's about as good as inbound gets: clear pain, approved budget, the right title. It's the easiest case we'll see, and the first version of the tool still won't do much with it.
Setting up
The code in this series is written against these versions. They're current as I write this, and they're pinned, for the same reason I'll argue for pinning the model later: when behavior changes, you want to know it wasn't the floor moving.
| Version | Notes | |
|---|---|---|
| Node.js | 24 LTS | 22 works too. The SDK supports any Node release still under maintenance, and Node 20 reached end of life in April 2026. |
@anthropic-ai/sdk | 0.131.0 | The official TypeScript SDK, and the only runtime dependency in this part. |
tsx | 4.23 | Runs a .ts file directly, with no build step. |
typescript | 7.0 | Type-checking only. tsx strips types without checking them. |
| Model | claude-opus-5-5 | Pinned by exact ID in every call. |
The project is an ordinary npm package:
mkdir lead-tool && cd lead-tool
npm init -y
npm pkg set type=module
npm install @anthropic-ai/sdk@0.131.0
npm install -D tsx@4.23 typescript@7.0 @types/node@24
type=module is the line that matters. It tells Node, and tsx, to treat every file as an ES module, which is what lets the examples use await at the top level. Leave it out and the first example fails before it sends anything.
// tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noEmit": true,
"skipLibCheck": true
},
"include": ["src", "evals"]
}
tsx doesn't need this file to run anything. npx tsc does, and that's what catches type errors. NodeNext is also why the imports in the examples say ./lead.js for a file called lead.ts: TypeScript wants the name the file will have once it's compiled, and tsx maps it back.
Last, put your API key in the environment as ANTHROPIC_API_KEY. The SDK reads it from there, so it never appears in the code.
One call, no ceremony
Here is the entire program. Run it with npx tsx src/v0.ts.
// src/v0.ts
import Anthropic from "@anthropic-ai/sdk";
import { SAMPLE } from "./lead.js";
const client = new Anthropic();
const response = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
messages: [
{
role: "user",
content: `Score this sales lead from 1 to 10.
Name: ${SAMPLE.name}
Email: ${SAMPLE.email}
Company: ${SAMPLE.company}
Title: ${SAMPLE.title}
Notes: ${SAMPLE.notes}`,
},
],
});
for (const block of response.content) {
if (block.type === "text") console.log(block.text);
}
Three things here are worth naming, because they're the whole API surface and everything else in the series is built on them.
model is which Claude you're talking to. I'm using claude-opus-5-5 throughout. max_tokens is a hard ceiling on the response — the model doesn't know about it and just gets cut off mid-sentence if it runs long, which is a failure you'll meet at least once. The ceiling also covers the model's thinking, which we'll get to, so it's set far above what a one-line answer needs. And messages is an array of turns, each with a role of user or assistant. Here there's one turn, from us.
The response comes back as content, an array of blocks rather than a string. Blocks have types — text, and later thinking and tool_use — so you have to narrow before you can reach for .text. TypeScript makes you do it, which is the right outcome even when it's annoying.
Run it and you get something like: "I'd score this lead an 8 out of 10. Strong signals: they've clearly articulated a specific pain point around pipeline observability, they have approved budget for this quarter, and the Director of Data Platform title suggests..."
A good answer, but unfortunately very hard to use for anything real. We can't really do anything with it. We can't sort by it, route on it, put it in a database column, or compare two versions of this prompt to see which is better, because the output is prose and prose doesn't compare. String(response).includes("8") is not a plan. The number isn't even reliably a number — sometimes it's "8/10," sometimes "an 8," sometimes "somewhere between 7 and 8."
A more insidious problem — the model scored this lead an 8 against its own private idea of what a good lead is. Nowhere did we tell it what we sell, who we sell it to, or what disqualifies someone. It guessed well from context. We're one weird lead away from finding out that its guess and our business don't agree.
So, the two failures from the top of the piece, both of them already here in the first example. The output has no structure and the task has no definition. We'll fix the definition first, starting with where in the request it belongs.
Where the instructions actually go
The usual prompting advice tells you to "give the model a role." It doesn't say where or how, so let's define that now.
A request has two places to put words: the top-level system parameter, and the content of a user message. The distinction isn't stylistic. It's this:
system is the part of the prompt that is the same on every request. user is the part that is different on every request.
The rule is simple, and almost everything people say about system prompts follows from it. The instructions, the definition of the job, the criteria, the output contract, the examples — those are properties of the application, identical for lead 1 and lead 4,000. They belong in system. The lead itself is the input. It belongs in user.
There are three reasons to pay attention to this split, and each one gets more important as the series goes on:
- It's the visible interface boundary. The system prompt is your function body; the user message is your argument list. When you mix them, you get a function that takes its own source code as a parameter, and debugging it is miserable.
- The model treats system content as instruction from the operator and user content as data from a user. This distinction is not a security boundary — Part 3 is largely about what happens when hostile text arrives in a
userslot. - Everything before the first byte that changes can be cached, which is how you stop paying full price for the same 900-token preamble on every call. That's Part 2's problem, but the shape of the fix is decided here.
Here's the same call with the two halves separated, and with an actual definition of the job:
// src/v1.ts
import Anthropic from "@anthropic-ai/sdk";
import type { Lead } from "./lead.js";
const client = new Anthropic();
const SYSTEM = `You qualify inbound sales leads for a data-pipeline observability product.
What we sell: monitoring and alerting for data pipelines — Airflow, dbt, Spark, and
warehouse jobs. We detect broken and silently-degraded pipelines before the people
downstream do.
Who we sell to:
- Data platform teams: the people who own the orchestrator and the warehouse.
- Companies with roughly 200+ engineers. Smaller companies do not have a data platform
team and cannot justify the price.
- Average contract value is about $60,000/year.
- The buyer is a Director of Data Platform, Head of Data Engineering, or a VP with
those words nearby.
What disqualifies a lead:
- Students, hobbyists, and job seekers.
- Companies with no data platform to speak of.
- Consultants and agencies asking on behalf of an unnamed client.
- Competitors doing research.
- Anyone asking about a product we do not sell.
Score each lead 0-100 on how likely it is to become a $60k/year customer this year.`;
function userMessage(lead: Lead): string {
return `Qualify this inbound lead.
Name: ${lead.name}
Email: ${lead.email}
Company: ${lead.company}
Title: ${lead.title ?? "(not provided)"}
Notes: ${lead.notes}`;
}
export async function qualify(lead: Lead): Promise<string> {
const response = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
system: SYSTEM,
messages: [{ role: "user", content: userMessage(lead) }],
});
const text = response.content.find((b) => b.type === "text");
return text?.type === "text" ? text.text : "";
}
The output is still prose. Note what the system prompt is doing, because it's not "be specific" in the abstract. Every line in it is a decision somebody at this company actually made, and one the model had no way to infer: the 200-engineer floor, the $60k ACV, consultants being out of scope, the fact that a Marketing Ops Manager at a huge company isn't our buyer. Those are the facts that define our qualification.
That's the practical content of "be specific": state the task the way you'd state it to a competent new hire on their first morning — someone smart, who reads well, and who has never heard of your company. Everything you'd have to tell that person is something the model needs too. Everything you wouldn't bother telling them ("be thorough," "use your expertise") is decoration.
The model does not remember the last call
Before we go further: a fact that everything else in this series depends on, and that almost every later confusion traces back to.
Each call to messages.create is completely independent. The model retains nothing between calls. Nothing.
When I first read this, I found it hard to believe. Chat interfaces are built to hide this. So let's watch it fail:
// src/no-memory.ts
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
async function ask(content: string) {
const r = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
messages: [{ role: "user", content }],
});
const t = r.content.find((b) => b.type === "text");
return t?.type === "text" ? t.text : "";
}
console.log(await ask("Score the lead from Northwind Logistics 0-100. Answer with just the number."));
console.log(await ask("What score did you just give?"));
The second answer isn't a worse answer to the question. It's an answer to a question about an event that, from the model's side, never happened. There is no "just." There is no session. The second call arrives at a server with no idea that the first one occurred.
The fix isn't a setting. You send the history yourself:
const messages: Anthropic.MessageParam[] = [
{ role: "user", content: "Score the lead from Northwind Logistics 0-100. Answer with just the number." },
];
const first = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
messages,
});
messages.push({ role: "assistant", content: first.content });
messages.push({ role: "user", content: "What score did you just give?" });
const second = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
messages,
});
You push the model's own reply back into the array as an assistant turn, then push the new question. Now the second call can answer, because the answer is sitting right there in its input.
The message array you construct is the entire memory the model has.
Not most of it, all of it. If a fact isn't in the system prompt, the messages, or a tool result you appended yourself, then as far as this call is concerned it doesn't exist. Every capability that feels like memory is someone assembling that array on purpose.
For the lead qualifier this is a good thing — each lead is scored independently, which is exactly what we want, and there's nothing to remember. In Part 2 the array becomes the agent's working memory and starts growing every turn, and managing it becomes the job. In Part 3 we find out what happens when something poisonous gets into it. In Part 4 we give it a memory that outlives the process. The whole arc of this series is really about who is responsible for that array, and the answer never stops being "you."
A role is information
Now we can talk about role-based prompting without hand-waving, because we know where a role lives: it's the first line of system.
Our system prompt opens with "You qualify inbound sales leads for a data-pipeline observability product." That's a role — it tells the model what frame to read the input in, which changes what it notices. Read as a qualifier, "I have budget approved for this quarter" is the most important sentence in Priya's note; read as a support agent, it's small talk before the actual question.
What role prompting doesn't do is get better results because you were nice about it. Compare:
You are a world-class, elite sales development representative with 20 years of
experience and impeccable judgment.
against:
You qualify inbound leads for a $60k/year data observability product sold to data
platform teams at companies with 200+ engineers. You are stingy with high scores:
most inbound is not qualified, and a false positive costs an AE an afternoon.
The first one is a compliment. The second one contains four facts and a stated cost asymmetry, and it will change the output — the "stingy with high scores" clause in particular, because it tells the model which direction to err in, which is a genuine piece of business information that nothing in the input could supply.
The useful test: does this sentence tell the model something it could not have known? If yes, it's a role. If it only asserts that the model is good at things, delete it. It costs tokens and changes nothing.
Let's add the stance, and one more line we'll come back to. The prompt is about to be more than one string, so from here on it gets its own file, split into the pieces we'll keep adding to:
// src/prompt.ts
import type { Lead } from "./lead.js";
const ROLE = `You qualify inbound sales leads for a data-pipeline observability product.
You are stingy with high scores. Most inbound leads are not qualified, and a false
positive costs an account executive an afternoon. When you are unsure, score lower.`;
const CRITERIA = `What we sell: monitoring and alerting for data pipelines — Airflow, dbt, Spark, and
warehouse jobs. We detect broken and silently-degraded pipelines before the people
downstream do.
Who we sell to:
- Data platform teams: the people who own the orchestrator and the warehouse.
- Companies with roughly 200+ engineers. Smaller companies do not have a data platform
team and cannot justify the price.
- Average contract value is about $60,000/year.
- The buyer is a Director of Data Platform, Head of Data Engineering, or a VP with
those words nearby.
What disqualifies a lead:
- Students, hobbyists, and job seekers.
- Companies with no data platform to speak of.
- Consultants and agencies asking on behalf of an unnamed client.
- Competitors doing research.
- Anyone asking about a product we do not sell.
Score each lead 0-100 on how likely it is to become a $60k/year customer this year.`;
export const SYSTEM = [ROLE, CRITERIA].join("\n\n");
export function userMessage(lead: Lead): string {
return `Qualify this inbound lead.
Name: ${lead.name}
Email: ${lead.email}
Company: ${lead.company}
Title: ${lead.title ?? "(not provided)"}
Notes: ${lead.notes}`;
}
Only ROLE is new. CRITERIA and userMessage are carried over from src/v1.ts unchanged.
"When you are unsure, score lower" is the kind of line that sounds obviously right. We'll measure it later.
Reasoning before the verdict
Ask the model for a number and it will give you a number. Ask it for its reasoning and then the number, and the number gets better. This is chain-of-thought prompting. The mechanism behind it is worth understanding, because the mechanism is what tells you the one detail that actually matters.
The model generates tokens left to right, and each token is conditioned on everything before it. So if the first thing it emits is 87, that score was produced in a single forward pass with no intermediate work — a hasty judgment. Anything it writes afterward isn't the reasoning that produced the 87. It's a justification built to be consistent with a number that already exists. It may be fluent and sound like analysis, but it will almost never conclude "actually, on reflection, 40."
Flip the order and the score is conditioned on the analysis. The tokens that work through "900 employees, so above our floor; owns Airflow and dbt, so a real data platform; budget approved, so a real timeline; but also evaluating two competitors" are in the context by the time the score is generated. The model is now doing the thing we wanted rather than the thing that resembles it.
So: reasoning first, answer second. Not as a formatting preference. If you take one detail from this section, that's it.
Which brings me to a thing that has changed. On the model I'm using here, extended thinking is always on: the model does internal reasoning work before it produces a response, whether or not you asked it to, and there's no switch to turn it off. You can see a summary of it if you want:
const response = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
system: SYSTEM,
thinking: { type: "adaptive", display: "summarized" },
output_config: { effort: "high" },
messages: [{ role: "user", content: userMessage(lead) }],
});
for (const block of response.content) {
if (block.type === "thinking") console.log("[thinking]", block.thinking);
if (block.type === "text") console.log(block.text);
}
thinking: { type: "adaptive" } lets the model decide how much to think per request. display: "summarized" is what makes it visible — the default returns the blocks with empty text, so if you don't set it, thinking looks from the outside like a long pause. output_config.effort (low through max) is the dial for how much reasoning depth and token spend you're willing to buy. On Opus 5.5 it's the only dial, and it defaults to medium.
So a fair question is whether asking for reasoning in the prompt still buys anything. It does, but for a different reason than it used to, and the honest version is:
- Extended thinking is for quality. It's scratch work. It isn't returned raw, you shouldn't build on its exact contents, and it doesn't go in your database.
- Asking for reasoning in your output is for you. A
reasoningfield is a durable, loggable, human-readable record of why this lead got this score. When a salesperson says "why on earth is this a 90," that field is the answer. When you're debugging a batch of bad scores, that field is where you find out that the model has decided@gmail.commeans "not a real company" — a rule nobody wrote down, and one that's wrong maybe a fifth of the time.
Reasoning you can read is an auditable artifact — keep it first, so it's doing work rather than writing its own alibi.
Now let's fix the output.
A score you can parse
The naive version of this is to ask nicely:
Respond with JSON: {"score": <0-100>, "tier": "hot"|"warm"|"cold", "reasoning": "..."}
This mostly works, which is the problem. Mostly-works means that a few percent of the time you get a friendly preamble before the brace, or a Markdown fence around it, or a trailing "Let me know if you'd like me to elaborate!", and your JSON.parse throws at 2am on a lead that would have closed. Every team that has done this ends up with the same little stripFences() helper, the same regex, and the same bad memories.
Don't do that. The API can constrain the output to a schema:
// src/qualify.ts
import Anthropic from "@anthropic-ai/sdk";
import type { Lead } from "./lead.js";
import { SYSTEM, userMessage } from "./prompt.js";
const client = new Anthropic();
export interface Qualification {
reasoning: string;
company_size_estimate: string;
disqualifiers: string[];
score: number;
tier: "hot" | "warm" | "cold";
}
const SCHEMA = {
type: "object",
properties: {
reasoning: {
type: "string",
description:
"Work through the fit before scoring: company size, whether they have a " +
"data platform team, seniority of the contact, urgency, and any disqualifiers.",
},
company_size_estimate: {
type: "string",
description: "Your best estimate of engineering headcount, or 'unknown'.",
},
disqualifiers: {
type: "array",
items: { type: "string" },
description: "Any disqualifying facts found. Empty array if none.",
},
score: {
type: "integer",
description: "0-100. Likelihood of becoming a $60k/year customer this year.",
},
tier: {
type: "string",
enum: ["hot", "warm", "cold"],
description:
"hot: route to an AE today. warm: nurture. cold: do not contact.",
},
},
required: [
"reasoning",
"company_size_estimate",
"disqualifiers",
"score",
"tier",
],
additionalProperties: false,
} as const;
export async function qualify(lead: Lead): Promise<Qualification> {
const response = await client.beta.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
betas: ["server-side-fallback-2026-07-01"],
fallbacks: "default",
system: SYSTEM,
messages: [{ role: "user", content: userMessage(lead) }],
output_config: {
effort: "medium",
format: { type: "json_schema", schema: SCHEMA },
},
});
if (response.stop_reason !== "end_turn") {
throw new Error(`${lead.id}: stopped early (${response.stop_reason})`);
}
const block = response.content.find((b) => b.type === "text");
if (block?.type !== "text") throw new Error("no text block in response");
return JSON.parse(block.text) as Qualification;
}
output_config.format with a json_schema is the mechanism. The response is constrained during generation, so the text block is JSON — no fence, no preamble, no apology. additionalProperties: false is required on every object in the schema, and the first request with a new schema pays a small one-time compilation cost before it gets cached for a day.
The guarantee only holds if the model finishes. A response that hits max_tokens is half an object, and a refusal may not match the schema at all, so stop_reason gets checked before JSON.parse ever sees the text.
Two other lines in that call are new. effort: "medium" is already the default on Opus 5.5, so setting it changes nothing today. It's there so a change to the default can't quietly move your eval numbers, which is the same reason the model is pinned.
fallbacks: "default" is a beta feature, which is why this call goes through client.beta.messages and names a beta in betas. Opus 5.5 runs safety classifiers on every request, and they can decline one, which comes back as a refusal. With fallbacks on, the API reruns a declined request on another model inside the same call, picked according to why it was declined, and returns that answer instead. For a lead scorer that's insurance against one false positive leaving a lead unscored. If the fallback declines too, stop_reason is still refusal and the check above catches it.
The cost is that an answer can now come from a model you didn't name. response.model says which one actually answered, and during an eval it's worth logging, because a different model is exactly the confound pinning the model was supposed to remove.
Let's go through some things about the schema:
Field order is reasoning-first, on purpose. The object is generated in the order the schema declares, so reasoning is written before score exists. This is the chain-of-thought lesson from the last section, expressed as a data structure. If you reorder these fields so score comes first — and people do, because it reads better in a database browser — you've quietly turned the reasoning field into a post-hoc justification. Same schema, same prompt, worse scores, no error message.
description is prompt, not documentation. Nobody but the model reads those strings. The reasoning description is doing the same job a paragraph of instructions would, and it's doing it at the exact point of use. This is a small preview of the single highest-leverage lesson in Part 2, where the description field on a tool is the entire difference between an agent that works and an agent that flails.
disqualifiers exists to make a judgment explicit. We could have left it out and trusted the reasoning prose to mention disqualifiers. Giving it a field forces the model to commit to a list, which means we can now write if (q.disqualifiers.length > 0) in ordinary code and stop asking an LLM questions we can answer ourselves.
score is an integer with the range in its description, not in the schema. The constrained-decoding schema doesn't support numeric bounds like minimum and maximum, so nothing stops the model returning 4,000. In practice the description handles it. In principle, the schema constrains structure, and structure is not sense — semantic constraints are still your prompt's job, and validating them is still your code's job. Assume you'll have to check.
One convenience worth knowing: if you're already using Zod, client.messages.parse() with zodOutputFormat(YourSchema) will generate the schema and hand you back a validated object on response.parsed_output. It also closes the gap above. Write z.number().int().min(0).max(100) and the SDK takes the bounds out of the schema it sends, moves them into the field's description where the model still reads them, and checks them itself when the response comes back. A score of 4,000 throws instead of reaching your database. I'm writing the schema by hand here because I want it to be spelled out, and because it's one fewer dependency you'll have to deal with.
So now we have a function. qualify(lead) takes a Lead and returns a Qualification. It has a type. It can be tested, called in a loop, and written to a database. The lead tool has stopped being a chat toy and become a component — and that happened the moment the output got a structure, not when the prompt got better.
Three examples
Run the qualifier across twenty real leads and a specific kind of disagreement shows up. Not nonsense — the model isn't confused. It just doesn't draw the lines where we would.
A four-person consultancy asking on behalf of an unnamed client comes back as warm, because "consultants and agencies" is in the disqualifier list but the model reads the note as a legitimate buying signal and splits the difference. An eleven-person startup with genuinely acute pipeline pain comes back at 55, because our prompt says 200+ engineers but their note is the most convincing description of our problem in the whole batch. A staff engineer at a 2,300-person company who is "exploring options for next year" comes back hot, because everything about the fit is right except the one thing that actually matters, which is that nobody is going to buy anything for nine months.
The instinct is to write more prose. Add a paragraph about how to weigh urgency against fit. Add a sentence clarifying that consultants are really and truly excluded. That can work. It's also how prompts turn into 2,000-word documents that contradict themselves in three places.
The word doing too much work here is "qualified." It's an abstraction over a judgment that the people on our sales team make dozens of times a week and have never written down. Trying to define it in adjectives is trying to specify a decision boundary with a thesaurus.
Show examples instead. Three labeled leads, spanning the boundaries you actually care about:
// src/prompt.ts — add below CRITERIA, and replace the SYSTEM line
const EXAMPLES = `Here are three leads scored the way we want them scored.
---
Lead: Marc Boileau, Principal, Boileau Data Consulting (4 people)
Notes: "Evaluating observability tooling for a client. Can you send pricing?"
Score: 15, tier: cold
Disqualifiers: ["consultancy asking on behalf of an unnamed client"]
Why: A four-person consultancy is not the buyer, and we cannot qualify a client we
cannot see. Not hostile, just not a deal. If they name the client, that client is a
new lead.
---
Lead: Aisha Kone, Staff Data Engineer, Vellum Health (~2,300 employees)
Notes: "We're mapping out our data platform roadmap and exploring options for next
year. Do you have a technical overview?"
Score: 45, tier: warm
Disqualifiers: []
Why: Right company, right team, plausible eventual deal — but "next year" and a Staff
Engineer means there is no budget and no buyer in this conversation yet. Real, not now.
Warm is what "real, not now" looks like.
---
Lead: Henrik Sandvik, VP Data, Aurora Retail Group (~4,000 employees)
Notes: "A silent dbt failure cost us a full day of reporting during our quarter close.
My CFO asked me what we're doing about it. I'd like a call this week."
Score: 92, tier: hot
Disqualifiers: []
Why: Size, team, seniority, and a named incident with executive attention. The
timeline is set by someone other than us, which is the strongest signal inbound
produces.`;
export const SYSTEM = [ROLE, CRITERIA, EXAMPLES].join("\n\n");
Notice what each one is for. The first draws the line on consultants, which prose failed to draw. The second defines warm as a specific situation — "real, not now" — rather than a vague middle. The third shows that the top of the range is reserved for urgency someone else imposed, not merely good fit. All three are boundary cases, picked because they're where the model was actually wrong.
That's the difference between few-shot examples that help and few-shot examples that are useless. Three examples in the middle of the distribution teach nothing, because the model already gets the middle right. Pick the cases you'd argue about.
These go in system, after the criteria and before the request, for two reasons. They're part of the job definition, not part of this lead, so structurally they belong there. And they're stable bytes at the front of the prompt, which is what makes them cacheable (which we'll explore in Part 2).
A caution, because few-shot has a failure mode that's easy to miss: the model will copy the shape of your examples, including things you didn't intend as instruction. If all three of your examples have three sentences of reasoning, you'll get three sentences of reasoning forever, including on the lead that deserved a page. If your scores are 15, 45, and 92, don't be surprised when the output clusters near 15, 45, and 92. Examples teach distribution as well as criteria, so vary what you don't want copied.
How do you know any of that helped?
Everything above is defensible. I can argue for every change: the role carries real information, reasoning-before-answer follows from how generation works, the examples address failures the model actually made. It's a good story.
It's also, so far, only a story. I haven't shown you one piece of evidence that any of it improved anything, and a confident story about a prompt is exactly the kind of thing that turns out to be wrong. Prompting stops being a craft practice and starts being engineering at exactly the point where you can be wrong on purpose and find out cheaply.
So: a labeled set and a number.
The labeled set doesn't need to be big. It needs to be real, and it needs to be labeled by someone whose judgment you'd defer to — which for lead scoring means a salesperson, not you. Ten to thirty examples catches the failures that matter, and it's small enough that you'll actually build it. That beats the five-hundred-example set you never get around to.
Here are ten, labeled by our head of sales, saved as evals/leads.json:
[
{
"id": "L-1041",
"name": "Priya Raman",
"email": "priya.raman@northwindlogistics.com",
"company": "Northwind Logistics",
"title": "Director of Data Platform",
"notes": "We run about 40 Airflow DAGs feeding dbt into Snowflake. Right now the way we find out a pipeline broke is that finance tells us their dashboard is wrong. I have budget approved for this quarter and I'm looking at two other vendors.",
"label": "hot"
},
{
"id": "L-1042",
"name": "Kev",
"email": "kevbuildsthings@gmail.com",
"company": "n/a",
"notes": "doing a bootcamp capstone on data quality, is there a free tier?",
"label": "cold"
},
{
"id": "L-1043",
"name": "Marc Boileau",
"email": "marc@boileaudata.fr",
"company": "Boileau Data Consulting",
"title": "Principal",
"notes": "Evaluating observability tooling for a client. Can you send pricing?",
"label": "cold"
},
{
"id": "L-1044",
"name": "Dana Whitfield",
"email": "d.whitfield@continentalfreight.com",
"company": "Continental Freight",
"title": "Marketing Operations Manager",
"notes": "Heard you on a podcast, sounds interesting. What do you do exactly?",
"label": "cold"
},
{
"id": "L-1045",
"name": "Tomás Ruiz",
"email": "tomas@fernpost.io",
"company": "Fernpost",
"title": "CTO",
"notes": "Our pipelines break constantly and I find out from customers. We're 11 people but growing fast and this is my biggest operational headache.",
"label": "warm"
},
{
"id": "L-1046",
"name": "Aisha Kone",
"email": "akone@vellumhealth.com",
"company": "Vellum Health",
"title": "Staff Data Engineer",
"notes": "We're mapping out our data platform roadmap and exploring options for next year. Do you have a technical overview?",
"label": "warm"
},
{
"id": "L-1047",
"name": "Henrik Sandvik",
"email": "h.sandvik@auroraretail.no",
"company": "Aurora Retail Group",
"title": "VP Data",
"notes": "A silent dbt failure cost us a full day of reporting during our quarter close. My CFO asked me what we're doing about it. I'd like a call this week.",
"label": "hot"
},
{
"id": "L-1048",
"name": "sarah",
"email": "sarah@brightloom.io",
"company": "Brightloom",
"notes": "pricing?",
"label": "warm"
},
{
"id": "L-1049",
"name": "Owen Pace",
"email": "opace@stonebridgecsd.k12.us",
"company": "Stonebridge County School District",
"title": "IT Director",
"notes": "We need help getting our student data reporting to be accurate and on time.",
"label": "cold"
},
{
"id": "L-1050",
"name": "Ingrid Møller",
"email": "ingrid.moller@kestrelbank.com",
"company": "Kestrel Bank",
"title": "Head of Data Engineering",
"notes": "We have a Spark and Airflow estate across three teams, roughly 6,000 engineers company-wide. Our procurement cycle is long and we'd need SOC 2 and a security review. Interested but this will take time.",
"label": "hot"
}
]
Two of those deserve comment, because they're the ones that make the set useful. L-1045 is an eleven-person company labeled warm — our written criteria say 200+ engineers, and our head of sales overrode it on the strength of the pain. L-1050 is labeled hot despite a stated slow procurement cycle, because a 6,000-engineer bank with a named estate is worth waiting for. Both are places where the label disagrees with the prompt. Those are exactly the rows worth having. A labeled set that only contains cases your prompt already handles is a test suite of tautologies.
Now the runner. The metric needs a moment's thought, because "accuracy" is the obvious choice and the obvious choice is a bit thin here. Getting warm when the label is hot is a missed opportunity. Getting cold when the label is hot means a $60k deal sat in a queue nobody reads. Those are not the same size of mistake, and one number that averages them will hide the one you care about.
So: two numbers. Exact tier agreement, and a separate count of the mistake that costs real money.
// evals/run.ts
import { readFileSync } from "node:fs";
import { qualify } from "../src/qualify.js";
import type { Lead } from "../src/lead.js";
type Tier = "hot" | "warm" | "cold";
type LabeledLead = Lead & { label: Tier };
const leads: LabeledLead[] = JSON.parse(
readFileSync("./evals/leads.json", "utf8"),
);
const results = await Promise.all(
leads.map(async (lead) => {
const q = await qualify(lead);
return { lead, q, correct: q.tier === lead.label };
}),
);
const agreement = results.filter((r) => r.correct).length / results.length;
// The expensive mistake: a lead sales called hot, scored cold.
const missedHot = results.filter(
(r) => r.lead.label === "hot" && r.q.tier === "cold",
);
// The cheap-but-annoying one: cold lead sent to an AE.
const falseHot = results.filter(
(r) => r.lead.label === "cold" && r.q.tier === "hot",
);
console.log(`tier agreement: ${(agreement * 100).toFixed(0)}% (${results.length} leads)`);
console.log(`missed hot: ${missedHot.length}`);
console.log(`false hot: ${falseHot.length}`);
console.log();
for (const r of results) {
if (r.correct) continue;
console.log(`✗ ${r.lead.id} ${r.lead.company}`);
console.log(` label ${r.lead.label} → got ${r.q.tier} (${r.q.score})`);
console.log(` ${r.q.reasoning.split("\n")[0]}`);
console.log();
}
npx tsx evals/run.ts, about fifteen seconds, a few cents. The failure printout at the bottom matters as much as the number at the top — the number tells you whether to keep a change, the reasoning line tells you what to try next. An aggregate score on its own never tells you what to do next.
That's the whole practice. A tiny labeled set, a metric that separates the mistakes you care about from the ones you don't, and a loop you can run in under a minute. Part 3 takes this much further — LLM-as-judge for outputs with no single right answer, which is most of them once we're writing emails — but this is the foundation, and it's enough to find out whether the last hour of prompt-tuning was real.
TK — deferred, not forgotten. This section wants a table: agreement and missed-hot across the four prompt versions we built (naive → role → reasoning-first schema → few-shot). No numbers are going in here until someone runs it for real, and the prose around the table gets tuned to whatever actually happens — including the case where one of the four changes turns out not to help. Blocks publication; does not block drafting Parts 2-4.
Changing one thing at a time
With a number, the loop is boring in the way good engineering is boring: change one thing, run the evals, keep it or revert it.
One thing. Not the role and the examples and the wording of the score description. If you change three things and the number improves, you know something helped, and you'll carry the other two forward forever on the assumption they contributed. That's how prompts end up with paragraphs nobody can account for.
Now, the promised revert. Remember this line, from the section on roles:
You are stingy with high scores. Most inbound leads are not qualified, and a false
positive costs an account executive an afternoon. When you are unsure, score lower.
It carries real information, and I was pleased with it. But look at what the last sentence does to our labeled set. Two of our three hot labels are cases where the lead's fit is unambiguous but something else creates uncertainty: Kestrel Bank has a long procurement cycle, and Priya is actively evaluating two competitors. "When you are unsure, score lower" invites the model to read that uncertainty — will this close? — as a reason to downgrade the tier. But tier isn't a prediction of closing. It's a routing decision, and both of those leads should be routed to a human today.
That's a hypothesis, and it's testable in about a minute: delete the sentence, keep everything else, rerun. If missed hot drops and agreement holds, the sentence was costing us money and I revert it. If nothing moves, I learn that a line I was proud of does nothing, which is worth knowing and mildly humbling. If agreement drops, I keep it and I've learned that my analysis of my own prompt was wrong, which is the most valuable outcome of the three and the one nobody enjoys.
Two habits that make this loop survive contact with a real project:
Version the prompt, in the repo, in git. It's source code — the most behaviorally significant source code in the system, and the piece most likely to live in a Notion page that someone edits during a customer call. Diffs on a system prompt are as useful as diffs on anything else.
Write down why you kept each thing. A one-line comment next to a section of prompt saying "added after L-1045; agreement 70→80" is worth more than the section itself. Six months on, without it, nobody — including you — can tell which lines are load-bearing and which are superstition, so nobody dares delete anything, and the prompt only ever grows.
Three things worth knowing before you tune anything
Three short notes. Each one is a real trap that doesn't need a section of its own.
Determinism, and the knob that isn't there anymore. The old advice was to set temperature: 0 before measuring anything, so runs would be repeatable. That advice is now wrong in a specific way: temperature, top_p, and top_k have been removed from the current models, and sending them returns a 400. There's no dial. So the same prompt and the same lead can produce a different score on Tuesday, and your evals have run-to-run noise in them whether you like it or not.
You deal with this by getting repeatability everywhere else and being honest about what's left. Pin the model ID explicitly — never a floating alias — because a model change is the single largest confound available and it can otherwise happen to you without a deploy. Keep the prompt in version control so you know what actually ran. And treat a one-row swing on a ten-row eval as noise, not signal: if a change moves the number by a single lead, run it again before you believe it. This is genuinely a downgrade in convenience, and pretending otherwise doesn't help anyone. output_config.effort is the nearest thing to a control you still have, and it controls reasoning depth, not variance.
What this costs. Worth knowing precisely, because every tradeoff argued in the next three parts is a cost tradeoff. You can count tokens before you spend anything:
const count = await client.messages.countTokens({
model: "claude-opus-5-5",
system: SYSTEM,
messages: [{ role: "user", content: userMessage(lead) }],
});
console.log(count.input_tokens);
Our system prompt with the criteria and three examples runs somewhere around 900 tokens; a lead is 100 to 200; the response with reasoning is a few hundred. At Opus 5.5 rates — $4 per million input tokens, $20 per million output — that's about a cent per qualification, and about $100 for ten thousand leads a month. That's before thinking, which is billed as output and isn't in those numbers. (TK: measured cost per lead, thinking included, from the eval run.)
Two things about that arithmetic are worth holding onto. First, the system prompt is more than 80% of the input and it is identical every time, which is the entire reason prompt caching exists and why the "stable bytes first" rule keeps coming up. Second, the number is small now. In Part 2 the model starts calling tools in a loop and the whole conversation gets resent on every turn, so one lead becomes eight calls with a growing input. In Part 4, five agents each do that. Cost per lead goes from about a cent to something you'll notice, and the reason you should notice now is that "it's basically free" is true of exactly the architecture we're about to leave behind.
Where instructions go in a long prompt. Prompts are position-sensitive. Content at the very beginning and the very end of a long input gets attended to more reliably than content buried in the middle — which matters a great deal in Part 2, when we start pasting scraped web pages into the context. The practical rule: when you have a large blob of data and a short instruction about it, put the data first and the instruction after it. [10,000 tokens of company page] Now qualify this lead. beats Qualify this lead using the page below: [10,000 tokens]. The instruction in the second one is a long way from where the answer gets generated.
Where this leaves us
We have a function. qualify(lead) returns a typed Qualification with a score, a tier, an explicit list of disqualifiers, and readable reasoning that was generated before the score rather than after it. Behind it: a system prompt that contains our actual business, three examples chosen from real disagreements, ten labeled leads, and a runner that tells us in fifteen seconds whether the last change helped.
That isn't a prompt as a wish. It's an interface with a test suite, and living in a string instead of a function body doesn't exempt it from either of those words.
Now look at what it's actually reasoning about.
Name: sarah
Email: sarah@brightloom.io
Company: Brightloom
Title: (not provided)
Notes: pricing?
Our head of sales labeled that one warm — and the tool has no honest way to get there. There's no title, no headcount, no stack, no urgency. Sarah is a warm lead because a human recognized the domain, knew Brightloom is a 600-person company with a data team, and priced in that a one-word inquiry from that domain is worth a reply. None of that information is in the message array, and by now you know exactly what that means: as far as the model is concerned, it doesn't exist.
The best prompt in the world can't fix a missing fact. Every technique in this part makes the model better at reasoning over what it can see, and what it can see is a web form with four fields in it, most of them empty.
It needs to go find things out. That means giving it the ability to do things — look up a domain, read a company's website, query our CRM — and, critically, the ability to decide for itself which of those to do and when to stop. That's the step from a prompt to an agent, and it's one loop, about forty lines long.
That's Part 2.