Integrated Evals
Every skill answers: how would we know it works?
Most AI setups improve by feel. You change a prompt, it seems better, you move on. LifeOS builds the measurement in: every skill answers the eval question when it’s created, and the ones that can be measured, are.
Why it exists
You can’t climb a hill you can’t see. The whole system runs on verified iteration—current state to ideal state, checked at every step—and that only works when “better” is a number you can trust. Without evals, prompt changes are vibes. With them, a change either moved the score or it didn’t.
The other half is honesty about limits. Some work can’t be reduced to assertions—a piece of writing that has to land, a design that has to feel right. Forcing tests onto those produces fake rigor, which is worse than none. So the contract is a choice with a record: name a test suite, or decline with a written reason. The one hard requirement is that the question gets answered.
How it works
Evals here follow the same doctrine Anthropic publishes for its own agent evaluation. A test case is an input plus assertions about the output. Deterministic checks come first—they’re fast, free, and objective. A model judge grades only what code can’t reach, and it’s treated as debt: calibrated against your own ratings before its verdicts count, and steadily replaced as its disagreements with you get mined into deterministic checks.
The scores are real statistics. Every suite reports its pass rate with a confidence interval, reliability is measured as passing every trial rather than one lucky run, and comparisons run paired—the same cases through both versions—so a verdict that one prompt or model beats another actually holds.
Coverage grows on its own. Skill runs are logged deterministically, failures become new test cases, and the system’s integrity check reports which skills are measured, which declined, and which haven’t answered yet. The question is built into how skills get made, so coverage grows by default.