Is bigger the only direction?

In 2 lb ground beef, the number is not a quantity. In 2 milk, the number is a quantity. The two lines look almost the same.

A shopping list has both kinds of line. Telling them apart is most of what this post is about. Get it wrong and Aeyo — my shopping-list app — reminds you to buy a second package of beef you never asked for. The feature takes one typed line and turns it into four structured fields, thousands of times a day. It must not invent any of them.

The default answer to a problem like that is to reach for a bigger model. Is bigger really the only way to go?

I have fine-tuned models and written a great many prompts. My suspicion was that the default is a habit rather than a law — that for narrow, checkable, high-volume work, a small model is the right tool and not the consolation prize. But a suspicion is not a finding, so I built something that could embarrass me: 41 cases, two models forty times apart in size, three rounds each, and a scorer whose controls exist specifically to catch me flattering myself.

It embarrassed me twice. Those are rules 5 and 9.

Everything here is reproducible. The corpus, the scorer and every recorded round are public, and most of it re-runs offline with no account and no key.

The task, and why it is a fair test

Someone types one line into a shopping list. The app has to turn it into a structured item.

The last two rows are easy to underestimate. milk on its own has to come back untouched. A shopping list is mostly bare product names, so a field invented there is a false statement shown to someone about their own list. And milk and eggs has to produce nothing. Answering with one of them looks confident, and quietly throws half the input away.

I wrote 41 cases like these, with the answers I expected, then built a scorer to run them and count the results. Three things in it matter more than the case count:

• A floor. Nine cases whose correct answer is nothing — milk, shampoo, asdfgh. Failing these disqualifies a model whatever else it scores, because they are the most common input there is.
• Pairs. 2% milk and milk 2% must get the same answer. milk for Amy's birthday cake and bananas not too ripe must get different ones. A model that answers both members of a pair identically has learned nothing, and only the pair shows it.
• Held-out cases. The examples I give the model cover some kinds of qualifier and deliberately leave others out. That lets the score separate learned the operation from memorised my examples.

And scores are never collapsed into one percentage. Each field lands in one of five outcomes — exact, correctly absent, missing, wrong, or fabricated — because a wrong field costs more than a missing one, and one number cannot say so.

The result

Two models, forty times apart in size: granite-4.0-h-micro, about 3B, and gpt-oss-120b. Same corpus, same provider, same door. Two prompts: v1, written carefully in prose, and v5, the same prose plus eleven examples. Three rounds each.

With a prose-only prompt the 120B scored worse than the 3B. With the same eleven examples they land in the same place. The prose bars are each model’s single canonical round; the example bars are the median of three. Rule 9 shows the full spread.

Read the rows in pairs. With prose alone, both models score zero on the field the feature exists for, and the 120B is the worse of the two. With examples they are level, and the small model is ahead on that same field while costing roughly one-thirtieth as much.

⚠️ One honesty note about the middle. Between those two ends I changed two things, not one — the prompt and the JSON schema (rule 4 explains why the schema mattered). For the small model I have the rounds that separate them: the schema change alone moved that field from 0/18 to 6/18, and the examples took it from 6/18 to 14–15/18. For the 120B I do not have that split, because the one round that would have provided it lost enough calls to be inadmissible. So “examples did most of the work” is measured on the small model and inferred on the large one, and I would rather say that than let a clean-looking table imply otherwise.

That is the headline, and it is not the interesting part.

Nine rules, and what each one is worth

Each rule below carries its scope — that is the line under each heading. A finding measured on one model is worth much less than one measured on two, and pretending otherwise is how a team acquires folklore.

1. When a rule is ignored, show it once — do not rewrite the sentence

2 models · 4 rounds

My prose prompt did not merely fail to explain the rule. It stated the rule and named the failing lines verbatim: “NEVER a package size, weight, volume or percentage — 2 lb ground beef and 1 gallon milk have no quantity.”

Both models read that sentence and returned a quantity for both lines, every round.

Then I added one example — 12 oz vinegar → specification "12 oz", quantity empty — a line that appears nowhere in my tests. Both named lines were fixed, in all three rounds: 2/2. The same thing happened with “the line is ONE item, do not split it”: stated and ignored, shown and obeyed.

The 120B was not too small to follow the sentence. It was not reading the instruction as an instruction either.

Do this: when a rule is being ignored, resist rewriting it. Show it once, as an example. Rewriting prose that is already explicit is the most expensive way to discover the model was never reading it that way, and model size does not buy you out of it.

2. Examples teach the operation, not the cases — but only if you hold cases out

2 models · 7 rounds

The standard objection to few-shot prompting is that a stated rule generalises while examples just get pattern-matched. My own prompt asserted it. It did not hold.

The model did not memorise four examples. It worked out what the four examples were an example of.

Do this: teach on lines that are not in your test set, and keep at least one kind of case no example covers. Then the score means something. If untaught cases move, the model learned the operation. If only taught ones move, you have built an expensive lookup table, and your test has quietly stopped measuring anything. It costs nothing except deciding in advance.

3. An example set is a decision boundary — draw both sides of it

1 model · 3 rounds — the sharpest lesson here, and untested on the second model

I added 12 oz vinegar to teach a measure is not a count. It fixed the two measure cases. It also broke four counting cases that had been working: 2 milk, a dozen eggs, half a dozen eggs, eggs x2 all silently stopped counting.

Every case that survived had a container word in it — bags, boxes, cans.

The model had learned a real rule, consistent with every example I had given it: a number without a container word belongs in the description. My set contained a counting example, 4 cans of tuna, but it is container-worded — so nothing anywhere in the set showed a bare number being a count. It generalised correctly from one-sided evidence.

The fix was one more example: 2 lemons, a bare number that counts. The counts came back and the measures stayed fixed.

Do this: stop thinking of examples as a list of things to do. Every example places a point on one side of a line, and the model’s job is to work out where the line runs. Give it six points that all sit on the same side and it will draw a line somewhere you did not choose. For every distinction you teach, show an instance on each side.

4. An optional field is one the model may decline to think about

1 model · 7 rounds

For four rounds the small model scored 0/18 on extracting a specification. It simply never emitted the field. The field was optional in my JSON schema. Making it required-but-nullable — no other change, same prompt — took it to 6/18.

Do this: if a model is ignoring a field, look at your schema before you look at your prompt.

5. Never write “the model can’t do this” from a zero-shot result

2 models

There is a commit in my repository that says a model will not do something. It is dated August 22. It is written with some confidence. It is wrong.

What made me confident was not the failure — it was the controls. I had built an isolated probe that asked the model to do one thing and nothing else. Around the treatment I put two groups: a floor of lines that are already just the product, where the right answer is to hand them back unchanged, and a positive control of lines I already knew it could split.

The task was not too hard, the scoring was not broken, and the model was not confused about the format. It answered everything else perfectly and refused exactly the cases I cared about. The same behaviour had already shown up in three rounds of the full task.

That is what a clean result looks like. Four examples later it was 24 out of 24 — including 9/9 on kinds of phrase that appeared in none of the four.

My measurement was fine. The mistake was the sentence I attached to it. I had measured what one way of asking reaches, and I wrote down what the model can do. Those are not the same claim. One is about an instruction; the other is about a thing in the world.

Do this: write “under this instruction, it does not.” It is a clumsier sentence, and it is the only one you have evidence for.

6. Price an example set on measured total tokens — and never on someone else’s measurement

2 models · one of them contradicted the other

I had this filed as a general finding: examples pay for most of themselves. On the small model they do. The prompt grew from 442 tokens to 907, but generation shrank from 53 to 30 — the examples made the answers terser — so cost rose 40% rather than the 105% the prompt growth implied.

Then I measured the 120B, and it went the other way.

gpt-oss is a reasoning model: it spends completion budget before it emits anything, so examples cannot compress the part that dominates its bill. A direct model’s visible answer is the whole completion, and that is what examples shorten.

Do this: price examples on measured total tokens, on the model you are actually going to run.

7. Demonstrate what needs judgement; look up what needs knowledge

3 independent instruments agree

Two cases refused to learn from any example I wrote: a dozen eggs → 12 and half a dozen eggs → 6. My counting example used a digit, and word-numbers never followed it.

So I checked the same two cases three ways:

That is not a prompt problem. It is a fixed, finite mapping: the model either has it or it does not. The 120B has it because it has read more, not because it reasons better.

Do this: when a failure is lexical, the question is never which model knows this but is a table cheaper than the model that knows it. Here the table costs nothing and the model that knows it costs thirty times more. A model is a poor place to store a dictionary you could have written down.

8. Check whether you need a model at all

this task

Before giving the model credit for anything, I scored a purpose-built parser that contains no language model at all: ingredient-parser by Tom Strange (MIT) — a conditional-random-field sequence labeller trained on 81,000 recipe ingredient sentences, about 300 KB, 2 ms per line, no network, and byte-identical on every run.

⚠️ Read the next table knowing what I did to it. That parser was built to read “3 pounds pork shoulder, cut into 2-inch chunks”. I pointed it at shampoo, pencils, detergent and return library books — a domain it was never trained on, scored against a contract it was never designed for. Nothing below is a criticism of it, and if you have recipe text it will beat anything here.

It is not a candidate. It is a ruler. Counting turns out to be a parsing problem that barely needs a model at all — this thing beat a 3B at it. Deciding that double tips describes the pencil while for Amy’s new semester describes the errand is not a parsing problem, and that is exactly where it collapses: in recipe-land unsalted butter is the ingredient name, so it hands the line back whole. That gap is precisely what I am paying a model for.

⚠️ One disclosure on the 9/10. The parser reports amounts as it finds them; turning dozen into 12 and treating bag as a count rather than a weight took a small adapter that I wrote. Its lookups are mine, not the library’s, and granite gets 3 bags of chips right with no such help — so that column flatters the parser slightly. The adapter is published in three levels so you can see exactly what each one bought.

There is a second thing in that table worth sitting with. It fabricated nothing, ever, because when it does not know it returns the line unchanged. It also abstained correctly on every non-food item in the corpus — shampoo, toilet paper, return library books — which I did not expect from something trained only on recipes.

So it fails by silence, and an instructed model fails by assertion. Which of those you would rather ship is a product decision, not a benchmark one.

Do this: measure the dumb baseline first. It tells you which half of your problem actually requires a model, and it is usually less than you assumed.

9. One round is not a measurement

3 rounds each · the rule I learned last, by getting it wrong

I ran the 120B once under the good prompt, got 31/40 with zero fabrications, and wrote down that the two models fail in opposite directions: the small one invents field values but respects the output contract, the large one invents nothing but breaks it. A tidy opposition. I had it in a file within the hour.

Then I ran it twice more, and the story fell apart.

Round two fabricated four, including a quantity of 1 for the input milk — a count nobody typed, on the most common input there is. Round three fabricated one, kept the contract, and passed the floor cleanly. There is no opposition. My hypothesis had described a single sample and I had written it as though it described a model.

Here is what three rounds actually support, and it is a better finding than the one I lost:

On the score itself the two are indistinguishable. What separates them is consistency. The small model fabricated exactly three fields every round, on the same three cases. That is a bug with an address, and I can go and fix it. The large model fabricated zero, then four, then one, on different cases, and failed the floor twice for two different reasons.

For anything you intend to ship, that is the whole decision. A predictable failure is a work item. An unpredictable one on your most common input is not a tier you can deploy — and no single round tells you which of the two you have.

Do this: run it three times before you believe it. Two would have caught me. One is what I almost published.

What this does not establish

• One task, one provider, two models. This is a structured-output task with a checkable contract. I would expect the shape of these results to carry to similar tasks and I have not measured that.
• 41 cases, and my example sets have now been revised four times against them. That loop is tight enough that another revision would start measuring memory rather than generalisation.
• Rules 3 and 4 were measured on one model. They are the two I would most like to see someone else test.
• The 120B’s improvement is not cleanly attributed. Its two end-points changed prompt and schema together, and the round that would have separated them was lost to timeouts. On the small model the split is measured; on the large one it is inferred.
• Both models fail non-English input the same way — 2盒牛奶 comes back as milk from both. Unfixed, and the fix would cost me my only non-Latin test case.
• Nothing here says a small model is always the right choice. It says that on this task, the binding constraint was the instruction rather than the parameter count — and that once the instruction was fixed, the cheaper model was the steadier one.

Try it against your own task

The corpus, the scorer with its per-stage self-test, every recorded round with raw envelopes, the prompts, and the no-model baseline are at github.com/Dongli99/aeyo-parse-bakeoff, Apache-2.0.

Most of it runs without an account or key:

python3 line_scorer.py --self-test     # controls for every stage
python3 line_scorer.py --compare       # every recorded round, one table

The no-model baseline needs one install, and note that the PyPI name is not the repository name:

python3 -m pip install ingredient-parser-nlp
python3 crf_baseline.py --self-test && python3 crf_baseline.py M2

The scorer is the part I would actually steal. Three control groups, five outcome values instead of an accuracy number, and a self-test that fails if any stage stops being able to see its subject — because three times in this work an instrument property was published as a model property before anyone caught it.

The reason that isn’t cost

Cost is the obvious argument for a small model, and it is the weakest one available. The free tier was never the binding constraint here. First it was quality, then it was consistency.

The real reason is that Aeyo tries to use as little of a model as the job allows. Its learning uses no model at all — statistics, arithmetic, aggregation over what actually happened. Where a model helps is reading: turning the words you wrote into items on a list. It sees those words and nothing else Aeyo knows, it decides nothing, and it is there to save you typing rather than to learn about you.

Keeping that model small, cheap and predictable is not a budget decision. It is what makes the restraint structural instead of merely stated — and it is why rule 8, check whether you need a model at all, is the one I would keep if I had to throw the other eight away.

I call that thin AI. It deserves its own piece rather than a paragraph at the end of this one.