Idea

An estimate accurate enough to rank optimizations isn't accurate enough to validate one

Info

Originally written in French. Translated by AI — the meaning has been preserved, not the prose.

Main idea

A static audit reads a skill's content and the files it handles; it doesn't run it. Faced with a conditional branch, it takes the worst case; faced with a web call, it assumes a page of average size. The stated gap with reality is around fifteen percent.

To order steps whose shares run from two to twenty-one percent, that precision is amply enough: the ranking doesn't change. To assert that an optimization actually produced its gain, it is no longer enough — the measurement's uncertainty is of the same order as the effect to be demonstrated. Validating a before/after requires something other than an audit: instrumentation that records a run's real spending.

Why it matters

This fixes what you are entitled to conclude from an audit report, and what you cannot: it prioritizes, it doesn't certify.

And it guards against a frequent abuse — turning an estimate into a quantified commitment in front of a client or a management team.

Nuances and limits

When the expected gap far exceeds the margin of error — an optimization announced at minus forty percent — the comparison becomes conclusive again despite the imprecision.

And instrumenting has a cost of its own, which is only justified for the heaviest or most exposed skills.

Open questions

  • What level of instrumentation would be enough to settle a before/after without rewriting the skill being measured?