A cheaper model cost us every page, in a five-business test.

A small test

The step that writes a preview page for a qualified business had been built but never run against a real model. Everything upstream of it was tested — the crawl, the scoring, the refusals — but the part that writes English had only been exercised against fixtures. On 6 September it ran for the first time, over five businesses, with two different models on identical material.

Why the comparison was safe to run

Writing copy about a stranger’s business is the one place in the pipeline where a model could invent something that no measurement would catch. So the step is built to refuse: a statement that cannot be traced back to what the business itself published does not reach the page. That is what made it reasonable to try a cheaper model at all.

What the cheaper model did

On the same five businesses, the cheaper model produced no page that passed our quality check. The stronger model produced four. One business shows why: from the same source, the cheaper model found too little it could stand behind, so the copy that depended on it was stripped and the page was blocked. The stronger model found enough, invented nothing, and passed.

The failure was quiet, which is the point

Nothing wrong was written. The cheap run produced blocked pages and a record of what it had failed to support, because a page carrying an unsupported statement is not approved. That is the behaviour we wanted, and it is worth more than the saving: a cost decision degraded into fewer pages rather than into false statements about a business.

The one that both models failed

One business with almost nothing online failed under both models. No model fixes that, because the material does not exist — a known limit. For a business with that little to go on, a generated page is the wrong tool.

Five businesses is a test, not a finding about models in general. We would want a much larger run before treating the gap as a rate.