Main idea

Having a machine read books and standards and produce notes from them works on the first try. That is the easy part, and it is the part you anticipate.

What costs is the tuning. A first, permissive version detects too broadly: many notes, noise, insufficient quality. The next version tightens the filters, quality becomes excellent — and relevant notes, ones the permissive version had actually seen, disappear. No setting takes everything that matters without taking anything else; tuning means choosing which way the system will err.

That tuning does not fit in two lines of prompt. It takes the form of several dozen lines of instructions: a precise definition of each note type, validity criteria, rules for detecting regulatory references, a final terminology pass.

Why it matters

This overturns the split of effort you plan for at the outset. You budget for extraction and for proofreading the notes; in practice extraction is acquired immediately and the budget has to go to the rules.

And it points to where the asset is. Notes can be redone; the body of rules that took weeks to stabilize cannot — that is what has to be versioned, commented and defended.

Nuances and limits

The right threshold depends on what the base will be used for. Material meant to be reviewed every day tolerates less noise than material consulted on demand, where a weak note costs only one line too many.

And a threshold calibrated on a corpus of standards does not transfer as is to a corpus of opinion pieces: what the rules learn is as much the corpus as the criterion.

Open questions

• At what point does further tightening stop gaining more in quality than it loses in coverage?