ADR-0039: Durable derived values — one memoization mechanism with a lifetime per analytic

• Status: Accepted (decision gate B3.2 per ADR-0026; owner sign-off 2026-08-12). Supersedes ADR-0032, whose volatile memo becomes the :request lifetime of this ADR’s single axis.
• Date: 2026-08-12

Gate B3.2 — the asks

Added 2026-08-18 as the single retroactive application of ADR-0043, whose rule this ADR’s own history is the worked example for.

Source of the asks: feedback triage 2026-08-12, Q2 — the owner’s words, recorded before this ADR was written.

The second row is why ADR-0043 exists. The drop survived the batch, the agentic review closing act and the close-out, because every one of those holds the work against the ADR — and the ADR was internally coherent. A list of asks is the only artifact in the chain that could have caught it.

Context

The rule that made Portfolixir trustworthy is that every financial figure is reproducible from the transaction ledger (ADR-0004). It was implemented as something stricter: nothing derived may ever be kept. The product brief of 2026-08-12 (accepted as #663) names that over-reach and its two symptoms, and the PRD’s FR-1 was reworded on the same day to permit materialization under explicit conditions.

The symptoms are measured, not felt. Timing the existing daily performance walk (Portfolixir.Portfolios.Performance.analysis/2) against synthetic ledgers on a container comparable to a modest self-hosted box:

Two facts follow, and they are the whole case for this ADR:

• Cost grows about linearly with catalog size — 4× the securities cost 4.1× the time. Extrapolated to the reference volume named in the architecture document (500 securities, 10 years), a single walk lands around half a minute.
• The second call costs the same as the first. Nothing survives the computation, so every dashboard mount, every reload and every agent question pays full price again. This is the “repeat wait” that ADR-0032 addressed for the lifetime of a process and no longer.

(Measurement caveat, stated so nobody quotes these as budgets: they come from a synthetic ledger on CI-grade hardware, with weekly quote points rather than daily. They establish an order of magnitude — seconds to tens of seconds — not a target. The operator-facing number is produced by the command this ADR mandates in §6.)

Three decisions already touch this area and must be reconciled rather than ignored:

• ADR-0004 — holdings are derived, never stored. This ADR does not weaken it: a materialized value is a materialization of the single truth, not a second copy that could disagree.
• ADR-0032 — an ETS-backed memo of the daily walk, deliberately volatile, never surviving a restart.
• ADR-0035 — chose to remove redundant computation (one pricing pass per read) rather than to cache it.

The gate’s own framing required an argument against ADR-0035 rather than around it, and required that “everything is materialized” not be accepted as an answer by default.

Decision

1. One mechanism, three lifetimes — not three mechanisms

A derived value is a pure function value over a versioned, named basis. Everything else is a question of how long the result is kept. That yields a single axis with a lifetime parameter, not three separate designs:

ADR-0032 is superseded, not amended: its volatile memo becomes the :request case of this axis and stops existing as separate machinery.

2. Every registered analytic is eligible; a curated list is explicitly rejected

The four FR-1 properties (§4) are uniform — there is no argument that holds for one derived value and not another. Hand-picking which values may be materialized would encode today’s guesses into the design and would rot, exactly as the architecture document’s own structure section did.

Therefore: every analytic that registers a computation basis is eligible for any lifetime. The mechanism is uniform; the invariants are proven once against the mechanism rather than once per value; and which analytics actually run :durable is a configuration decision informed by measurement, not an architectural one.

This is the argued form of the gate’s “everything is not an answer” constraint. The constraint is real but it is about activation, not eligibility: materializing a value that computes in two milliseconds buys nothing and still adds an invalidation path that can be wrong. The cost of materialization is invalidation correctness, not storage — and invalidation is where systems of this kind fail (§5, I2/I3).

Initial activation: the daily performance walk, on the evidence in the Context section. Further activations are added when a measurement shows a wait, and each one is a configuration change plus its measurement, never a new mechanism.

3. ADR-0035 is inherited, not opposed

Removing a redundant call is strictly better than caching it: a removed call costs nothing, a cache costs invalidation. ADR-0035 stays the first line of defence, and materialization is admissible only for what remains after de-duplication. A story that proposes materializing a value which is simply computed more often than necessary is rejected in favour of computing it once.

4. The four binding properties (FR-1, restated as this ADR’s acceptance criteria)

Every materialized value must be:

1. rebuildable from transactions alone, with drop-and-rebuild a supported and tested operation;
2. versioned against the data-version counter, so staleness is detectable rather than suspected;
3. never silent about freshness — as_of plus an explicit stale marker, in the UI and in the API/MCP payload;
4. never authoritative for a write — no booking, import decision or consistency finding may read the derived layer instead of the ledger.

5. Invariants — the acceptance criteria in testable form

The defining property is an equation, so it is tested as invariants rather than as examples. All are blocking.

Two further conditions that are not invariants but block sign-off of the implementing story:

• Computation version in the key. The data-version counter covers data changes, not code changes. If a formula changes and the counter does not, the layer is silently wrong. The key is therefore (analytic_id, basis_hash, data_version, computation_version).
• Journal actor of a materialization write. A derived-value write is not a financial write and is not journaled; write_actor_test.exs must know that table class explicitly, because implicit non-coverage is how the strongest existing gate would be bypassed rather than weakened.

6. The rebuild budget is a measurement, not a guessed number

Drop-and-rebuild is the emergency procedure, and an emergency procedure with an unknown runtime is not one. But a budget invented before anything exists is how ADR-0032 and ADR-0035 came to be decided on a felt symptom.

Therefore:

• drop-and-rebuild is a single operator command, not a procedure;
• its runtime is measured on operator hardware on first run and recorded in this ADR as an amendment;
• the acceptance criterion until then is the shape, not the number: the rebuild completes unattended in one command, and reports its own runtime.

If the recorded number later proves intolerable, that is a finding with evidence behind it rather than a prediction.

Amendment (2026-08-14, C5 implementation). First measured run of mix portfolixir.derived.rebuild against the Context section’s synthetic dataset (50 securities, 1,001 bookings, 26,100 weekly quote rows, one portfolio, ~3,650 walk days): 3.9 s (3,861/3,865 ms across two runs, reported by the command itself; drop, version-log compaction and re-warming the operative scopes included). Hardware caveat: measured on the CI-grade container the Context timings come from, not on operator hardware — the operator-hardware number this section asks for is recorded on the first real run there and replaces this one as the reference. For calibration, the same dataset’s repeat-read timings after activation: 2,628.9 ms with the layer off, 2.4 ms from the in-memory tier, 24.5 ms in a cold process served from the stored row.

Amendment (2026-08-18, §4 of the 2026-08-15 amendment — issue #711). The activation measurement, taken with the committed command this amendment adds (mix portfolixir.derived.measure, which seeds a deterministic synthetic ledger and times each candidate with the layer off, twice — the same first-call / second-call shape as the Context table). Two sizes, same container class as the Context row; wall-clock milliseconds:

Four findings, and the third is the one that changes what this section is for.

1. The activation the measurement justifies is performance_view_analysis, and it is the only one. It costs the same seconds as the walk activated in the first pass, its second call costs the same as its first, and it is what the Wealth page and the dashboard card open — yet it was registered from the start and left at the :request default, so it died with every restart. Activated at :durable in config/config.exs, with its storage, its restart read and its I3 backdating case tested in test/portfolixir/derived/view_analysis_activation_test.exs.

2. Cost still grows about linearly with catalog size, reproducing the Context row from an independent seed: 4× the securities cost 3.7–3.8× the time on both walks.

3. Nothing else is a derived-value problem — that is a finding, not an omission. Two orders of magnitude separate the walks from everything else: at the larger size the most expensive non-walk candidate is 393 ms, 24× cheaper than the walk beside it. The operator’s report of “computing cues across the app” therefore has two different causes, and only one of them is this ADR’s: the walks, fixed by activation plus the write-triggered refresh of #710; and surfaces that load a 100–400 ms value asynchronously and show a cue while doing it, which is a rendering decision, not a lifetime decision. Filed as #723 rather than answered with an activation that would not have helped.

4. The non-walk candidates’ costs are not independent, so activating them individually would materialize four overlapping copies of the same work: valuation, allocation, the per-category result and the holdings view all sit on Ledger.holdings_for_portfolio/2, which is itself the largest of them. ADR-0035’s “remove the redundant computation rather than cache it” remains the first line of defence there, exactly as §2 says it should.

Same caveat as the Context table, restated so nobody quotes these as budgets: synthetic ledger, CI-grade container, weekly quote points. They establish an order of magnitude and a ranking, not a target. Re-run the command rather than citing these numbers on different hardware.

Consequences

Easier. The repeat wait disappears for whatever is activated: the walk that costs seconds today is paid once per invalidation rather than per mount. The agent reads finished figures instead of reconstructing them, which is what the agent-side criteria (≤ 5 calls for the weekly run, −70 % response volume) attach to. FR-39 through FR-42 — derived metrics per security and per view, contribution analysis, exposure decomposition — gain a place for their values to live, which is the dependency the requirement registry records against them.

Harder. Every activated value needs a correct invalidation path, and the backdated-transaction case (I3) is the one that naive implementations get wrong. The test suite grows a property-based layer that is slower than example tests and harder to debug when it fails. Freshness becomes part of every payload the layer touches, so the API and MCP surfaces change additively and the contract fixtures (architecture D5) become more valuable than they already are.

Off-limits. The derived layer may never be read by a write path (I7), never be the answer to a consistency question, and never outlive a formula change silently (computation version in the key). A curated list of “cacheable values” is off the table by construction — if a value cannot satisfy the four properties, the answer is that it is not a derived value at all. The historical-exchange-rate case is precisely that: a rate observed on a past day is an observation, not a derived value, and conserving it is an input-capture concern upstream of this ADR, not a reason to weaken “rebuildable”. This ADR does not decide that capture; it depends on it for I6’s fixture.

Accepted trade-off. Uniform eligibility means the mechanism must be right for values nobody has thought about yet, which is a higher bar than making one cache work for one series. That is deliberate: the alternative is a list that ages, and this repository has just spent a validation pass documenting what an aging list costs.

Amendment: refresh is triggered by the write, not by the next reader (2026-08-15)

What the operator observed

“If we already have all the numbers computed, why am I still shown a ‘computing’ cue for a second, on all sorts of figures? They should not be computed when the UI asks for them — they should be held ready and refreshed automatically.”

Three separate causes, all real, none a malfunction of the mechanism above.

1. Almost nothing is activated. Registry holds two analytics (performance_analysis, performance_view_analysis) and config.exs activates exactly one at :durable. Every other figure on the human surface — valuation, allocation and drift, holdings, income, tax, risk, the snapshot comparison, the securities metrics — is not registered at all, so it is recomputed on every read exactly as before this ADR. §2 deliberately made activation a configuration decision informed by measurement; the measurement was never taken past the first analytic, so “eligible for any lifetime” has stayed theoretical for everything but one value.

2. The durable value is refreshed by its next reader. Warmup has two triggers — boot and day rollover — and neither is a write. So the sequence after any booking is: the write bumps the data version, the entry becomes stale, and the first person to open a page pays the recomputation synchronously and watches the cue. “Paid once per invalidation rather than per mount” is true and is not what the operator asked for.

3. That difference was the point of the gate, and it did not reach the decision. The B3.2 gate was opened on two asks (feedback triage 2026-08-12, Q2): durability, and push instead of pull — recomputation triggered by the write that invalidated it, or by a schedule. This ADR decided the first and is silent on the second. Naming it plainly: the gap is in the decision text, not in the implementation of it.

Decision

1. Refresh is scheduled by the invalidation, not by the next read. When a write bumps a basis’s data version, the operative scopes for that basis are re-materialized in the background, through the same request path Warmup already uses (§ “it warms through the same API a request uses” — there must not be a second computation path). A reader arriving before the refresh finishes is served the superseded value through peek/3, labelled, exactly as §6 already prescribes; a reader arriving after it is served a fresh value with no cue at all.

2. Refresh is coalesced, and this is the load-bearing part. A naive “recompute on every bump” is worse than the pull it replaces: a Portfolio Performance import bumps the version per booking, and BlastRadius widens most resource types to :all, so a single import would queue thousands of full recomputations of every scope. Therefore:

• bumps are collected per basis and drained after a quiet period, so a burst of writes produces one refresh;
• a refresh already running for a basis is not duplicated; a bump arriving during it marks the basis dirty again and re-queues once;
• the queue is bounded and drops to “recompute this basis fully” rather than growing per-write work.

Import remains the worst case by construction and is the acceptance scenario: importing a large export must produce a single refresh per affected basis, not one per row.

3. Never a correctness dependency. The background refresh is an optimisation of when work happens. fetch/4 keeps its contract — a stale entry is recomputed on read — so a failed, slow or disabled refresher costs latency and never freshness. It is disabled by the same switch as the rest of the layer.

4. Activation is a measurement task with an owner, not an open option. The figures the operator actually waits on are enumerated, measured on a realistic data set, and assigned a lifetime; the result is recorded next to the §6 rebuild budget rather than left as configuration nobody revisits. An analytic that is cheap enough not to need a lifetime is recorded as such — that is a finding, not an omission.

Consequences

• The operator’s expectation is met for activated values: numbers are held ready and refreshed by the event that invalidated them.
• The cost moves from the reader to a background process, so the app does scheduled work it did not do before. Bounded by §2’s coalescing, and the budget is the same rebuild measurement §6 already requires.
• Risk-tier attention: the coalescing rules are where an import turns into a recomputation storm. The import scenario is a test, not a review note.