blob: dd3372eccb52f9b7d3e53b9f1ef6e48f28855a21 [file] [view]
# Transitioning the Go SIMD generators to `internal/spec`
This plan lays out how to transition the myriad generators and hand-written
sources of the SIMD API from disjoint, manually-enforced consistency, to a
single underlying executable API specification that provides all method
signatures, documentation, and checkable semantics.
See [README.md](README.md) for background on the SIMD packages, the "one name,
one meaning" rule, the generators, how simdgen works, what spec and specgen are,
and how to run things.
## Table of contents
- Part 1: Decisions and rationale
- Part 2: Task graph
- Part 3: Task details
- Part 4: Risks and unknowns
- Appendix A: Regenerating the two generated files
## The plan in brief
`internal/spec` — generic Go functions describing each operation — becomes the
single hand-written description of the SIMD API. Today the same operation is
described in six places, and they have already drifted from each other (§1.5).
Three things have to happen, and they are largely independent:
1. **Build the `specfill` checker.** One AST pass that injects spec's docs into
generated and hand-written source and verifies signatures against spec. Types
can be enforced almost at once — SPEC-STATS.txt §2 shows zero type-level
mismatches between spec and the shipped API.
2. **Write the spec functions** (`spec-common` and `spec-all`). Coverage is in
SPEC-STATS.txt §1. This is the bottleneck the other
tracks draw on.
3. **Retire the old inputs**: the other doc sources, then `categories.yaml`, at
which point the unifier no longer defines the API — it only expands one arch
entry into the machine instructions matching it.
Behavioral conformance against `internal/simdref` (track C) runs in parallel.
Beyond its own `ci-audit`, it depends only on spec coverage and on the `width`
and `impl-defined` decisions.
## Monitoring plan progress
Progress on this transition is tracked in two ways:
**Task completion** is tracked directly in this document via the `[ ]`
checkboxes on each task and risk. The task table in part 2 summarizes all task
completion in one place.
[SPEC-TASKS.dot](SPEC-TASKS.dot) shows a visual representation of the task graph
and completed tasks. It is generated directly from this document by running `go
run ./cmd/spectasks -w` from `_gen`. This generator also checks the consistency
of this doc, and the test in `./cmd/spectasks` enforces doc consistency in CI.
Rerun the generator after modifying any task metadata.
**Measurements** live in [SPEC-STATS.txt](SPEC-STATS.txt), which tracks spec
coverage of the API, signature and name agreement, documentation sources, and
retirement of the old inputs. It is generated by running `go run ./cmd/specstats
-w` from `_gen`. Every figure in that file is a fraction heading for 100% or a
count heading for zero, tagged with the tasks that move it. Indented, untagged
numbers are context breaking down the figure above them. Part 1 of this doc
quotes measurements that motivated specific decisions, stamped with the CL at
which they were measured; those are frozen and do not need refreshing.
## How to implement a task
Part 2 tells you which task to pick up: the next one whose prerequisites are all
ticked. [SPEC-TASKS.dot](SPEC-TASKS.dot) draws the same thing if a picture
helps. The commit sequences in Part 3 are guesses made before the work started;
deviate when the code tells you to, but say so in the CL description.
Tasks and risks carry `[ ]` checkboxes. When you resolve one, tick the box,
update the task text to reflect the reality of the implementation so it remains
useful to reference when performing follow-on tasks (you can drop or summarize
the exact commit sequence), and regenerate [SPEC-TASKS.dot](SPEC-TASKS.dot).
Commit this as its own CL on top of the CLs that implement the task, and include
justifications and reasoning for any changes to the task text in the commit
message.
## Key concepts and terms
**`specfill`** is the tool this plan builds first and then leans on. It is one
AST pass over Go source that injects spec-derived doc comments and parameter
and result names, and verifies signatures against spec. It exists as both a
package that can be integrated into generators and a standalone tool:
- **`specdoc.Fill`** is the library entry point — Go source in, Go source out:
`func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`.
If verification checks fail, it returns a `*Report` as an error. Generators
call it on their `gentools` buffer before writing.
- **`cmd/specfill`** is the `gentools`-based command wrapping it, for running
over hand-written files. Where this document writes `specfill -w`, that is
the command; where it writes `specfill`, that is the library function. CI
enforcement is a `go test` in this package.
These package and command names are suggestions; the split between library and
command is not.
**Target** means one of amd64, arm64 (NEON), arm64 SVE, or wasm. See README.md.
**Spec operation** means one function in package `spec`. Each expands across
element types, widths, and templated names into many API names and many
`(receiver, name)` pairs, so it is the grain spec is *authored* at — and the
only grain at which "implemented nowhere" means anything (SPEC-STATS.txt §5).
**Operation** (unqualified) means the API-level operation a name denotes,
whichever of spec, `categories.yaml`, or generated source describes it. A
`categories.yaml` entry is called an *entry*; one operation name can have several.
**Granularity.** Spec coverage is measured at three granularities:
`(target, receiver, name)` — every exported declaration, counted once per build
configuration; `(receiver, name)` — collapsed across targets, the primary
denominator since spec describes an operation once; and `name` — collapsed across
receivers too, coarser and misleading when a name exists on different receiver
sets in spec and the API (SPEC-STATS.txt §5). SPEC-STATS.txt §1 reports all
three. `String` and `Len` are excluded throughout (§1.7).
---
# Part 1 — Decisions and rationale
These are the load-bearing decisions. Each was contested during design. The
reasoning is recorded so it need not be re-litigated, while also making it
easier to revisit decisions if new evidence arises.
Measurements in this section were taken at CL 831485 unless otherwise noted.
They should NOT be updated unless decisions need revisiting.
## 1.1 We are not rewriting simdgen
The temptation is real: simdgen has 28 stub templates whose *names* encode
arity, operand permutation, and immediate placement (`op3_231Type1`, `op4_31`,
`op2Imm8_II`); 42 fields across `RawOperation` (23) and `Operand` (19); and 27
bail-outs whose messages begin "simdgen only supports" or "simdgen does not".
But the layer accounting says the mess is small and the value is large (at
CL 831485; Go counts include `_test.go`; "signature derivation + emission" is
`simdgen/gen_*.go`, `godefs.go`, `arch.go`, and `types/operation.go`):
| Layer | Lines | Affected by "was unification wrong?" |
|---|---|---|
| Platform loaders | 5,100 | No — irreplaceable |
| Unifier engine | 3,169 | Partly |
| Signature derivation + emission | 4,715 | The ~500 messiest lines, yes |
| `categories.yaml` | 1,284 | Deleted by this plan anyway |
| `go_GOARCH.yaml` | 4,475 | No |
The part that would be thrown away if unification were the wrong call is the ~500
messiest lines of signature derivation, against irreplaceable platform loaders
several times that size.
A few thousand lines of hand-written YAML drive tens of thousands of lines of
generated API and compiler backend across four targets. That ratio is good.
What is *not* earning its keep is unification's role as **API definer** — the
`categories.yaml` ↔ `go_GOARCH.yaml` join — and this transition removes that role
by construction. The unifier's remaining job, expanding one arch entry into every
matching concrete instruction variant against the ISA database, it does well.
## 1.2 Spec is not a unifier input
Feeding spec into unification via `DefBuilder` is easy and wrong. Spec's
structure would be flattened into the unstructured value language the moment it
crossed the boundary, recreating the exact failure mode we are escaping. Instead,
simdgen consumes `[]*specgen.Func` as a typed Go table.
Furthermore, spec is inherently only aware of the structure of the Go SIMD API,
but the inputs to the unifier are in platform terms. Much of the output of the
unifier is information on how to map the platform structure into the Go SIMD
API, so we can't bring spec in until *after* that mapping has been processed.
## 1.3 What belongs in spec
The test is: **is it a checkable claim about the spec's own semantics?**
- `commutative` — yes. `Add(x,y) == Add(y,x)` is testable against the executable
body, and spec is the only place that *can* check it.
- `noGenericOps`, `noTypes` — no. Claims about generator output.
- `constImm` — no. A claim about x86 encoding.
- `regexpTag`, `signed` — no. Claims about the unifier.
The test is self-policing: anything that passes it falls out of the conformance
testing we want anyway. Note it admits facts the *compiler* consumes — the wrong
line to draw is "API vs compiler"; the right one is "intrinsic to the operation
vs incidental to the mechanism or platform."
For `commutative`, no name in `categories.yaml` currently needs
platform-specific commutativity, so a function-level directive in spec is
sufficient today. However, there may be corner cases we haven't uncovered;
notably, we have evidence that `Min`/`Max` on floats may not be commutative on
all platforms. It may suffice to simply not mark these operations as
commutative, and to document when an operation is not commutative for surprising
reasons.
Applying the test to `categories.yaml`'s fields (at CL 831485; 217 entries, 195
unique operation names, 163 exported):
| Field | Count | Transition plan |
|---|---|---|
| `go` | 217 | → spec function name |
| `commutative` | 217 | → `//specgen:commutative` directive; `categories.yaml` copy removed by `sig-from-spec` |
| `documentation` | 211 | → spec doc comment; removed by `template-docs` |
| `regexpTag` | 25 | unification join key; role unverified (see `risk-cat-unknowns`) |
| `constImm` | 22 | → sideways into `go_amd64.yaml` (x86 encodings) |
| `out` | 14 | join key used only for selecting a doc variant; → multiple spec funcs sharing one `//specgen:name` |
| `elemBits` | 14 | same |
| `bits` | 4 | same |
| `signed` | 12 | same |
| `noGenericOps` 13 / `noTypes` 6 | 13 entries | **open question** — compiler-only ops |
Task `cat-rehome` tackles most of the "transition plan" column.
`compiler-ops-impl` handles the `noGenericOps` / `noTypes` entries.
## 1.4 `specfill` parses; it does not type-check
`ops_amd64.go` and `types_amd64.go` have minimal imports, and no exported
signature in either package uses a qualified type, so `go/parser` is sufficient —
no `go/types`, no `packages.Load`, no GOEXPERIMENT wrangling, no cross-module
loading of GOROOT std packages.
This is better than a workaround. Type-checking sees one build configuration at a
time and would make the cross-architecture comparison *harder*. Parsing all files
regardless of build tag yields `(Int32x4) Add` from `ops_amd64.go`,
`ops_arm64.go`, and `ops_wasm.go` in a single pass, which is precisely the
comparison we want. `apisum` already does this, so it is proven on this API.
`cmd/specstats` (Appendix A) takes a different route to the same end: it loads
syntax through `go/packages` once per build configuration, because it reports
per-target figures. It still does not type-check, and `specfill` does not need
per-configuration loading.
## 1.5 Two-pass doc injection, and why the "disadvantage" is the point
The motivation is doc divergence. At CL 831485 there were **six independent
documentation sources** — `categories.yaml`, wasmgen inline strings, tmplgen
inline strings, `midway/comments.yaml`, hand-written `simd_emulated.go`, and
hand-written `archsimd` files — and none agreed with any other. All 90
`(receiver, name)` pairs documented in both `simd` and `archsimd` had different
doc text, and within the portable package, `simd_stubs.go` (midway-generated) and
`simd_emulated.go` (hand-written) had drifted on 40 of 336 mirrored methods.
`Add` shipped with three independent wordings:
| Where | Text | Source |
|---|---|---|
| `ops_amd64.go`, `ops_sve.go` | *Add adds corresponding elements of two vectors.* | `categories.yaml` |
| `ops_wasm.go` | *Add returns the result of adding x and y, elementwise.* | wasmgen, inline |
| `simd_stubs.go`, `simd_emulated.go` | *Add returns the element-wise sum of x and y.* | `midway/comments.yaml` |
124 method names are produced by more than one generator — the easiest place for
disagreement to creep in. simdgen accounts for the majority of declarations, but
tmplgen, wasmgen, midway, and hand-written code own the rest, and those sets
overlap — any instrument that only sees simdgen misses a substantial fraction of
the problem (SPEC-STATS.txt §1 has the per-generator counts).
Generators pass their generated code to `specfill`, which checks signatures and
injects docs before writing to disk; in the end state, generators emit
declarations and implementation notes (§1.6) but no API prose of their own. The
alternative — having specgen emit the whole declaration, as `refgen` does — was
rejected because it would make agreement with spec true by construction and
leave nothing to verify. A generator whose internal model of the API had drifted
would be silently papered over. Because the template still produces the
signature, `specfill` sees two independent witnesses to the same declaration and
can compare them: docs in, signatures verified, coverage reported, one AST walk.
Consequences:
- **`specfill` is an override, not a replacement.** Generators keep emitting
their current docs; `specfill` replaces the spec-owned paragraphs only when spec
knows the operation. Fallback is therefore automatic and no flag day is
required. Prose-less emission is the *end state* reached when a generator's doc
source empties, not a precondition.
- **wasmgen and tmplgen need only a call before flush**, plus making the notes
they emit follow §1.6's prefixes.
- **Do not hook this into `gentools.Files.Flush()`.** `gentools` is a general
utility; coupling it to specgen would make every consumer depend on
`golang.org/x/tools/go/packages` and a loadable spec tree. A narrow hook
mechanism in gentools plus a few lines per generator keeps the boundary.
## 1.6 The doc-ownership convention
`specfill` rewrites part of an existing doc comment, so it needs a rule for which
part it may touch, and running it twice must be idempotent. The obvious rule,
"everything above the first bare `//`", doesn't work. At CL 831485 only 2,759 of
3,920 documented declarations have that separator. And spec's own docs contain
blank comment lines: `Add`'s is a summary, a `//`, then `z[i] = x[i] + y[i]`.
Hence, using a bare `//` would not be idempotent.
**The rule.** A doc comment is spec-owned paragraphs, followed by
"implementation notes", followed by directive comments (`//go:fix inline` and
similar). Spec-owned paragraphs describe the behavior of an operation.
Implementation notes are details of how one target implements an operation. The
sections must appear strictly in this order, and the tools will validate this.
An *implementation note* is a paragraph that starts with one of a fixed set of
prefixes: `Asm:`, `CPU Feature:`, `Emulated[,\n]`, `Deprecated:`,
`Performance:`, or `Note:`. The match is exact and case-sensitive. Only the
first line of a paragraph is checked, so a note can combine prefixes (`Asm:
VPADDD, CPU Feature: AVX`) and wrap freely.
**Checks.** Strict ordering is necessary for idempotence, and the tools perform
the following checks to ensure we maintain it: (a) In an existing comment,
spec-owned paragraphs (those without an implementation note prefix) must appear
strictly before implementation notes (those with an implementation note prefix),
which must appear strictly before directive comments. (b) Spec's docs may not
contain a line starting with a note prefix, and spec stores them in canonical
`go/doc/comment` form.
**Rewrite.** `specfill` rewrites a comment only when spec has an entry for the
declaration and check (a) passes. The new comment is the spec doc, then a blank
line if there are implementation notes, then those notes unchanged, then the
directives unchanged. In every other case the comment is left alone. This is
idempotent: by (b) the injected spec doc never looks like a note, so the next run
splits the comment at the same place and produces the same bytes. Idempotence
can't catch text lost on the *first* run, though, so as we transition to spec,
generated docs for newly covered operations must be reviewed carefully.
**Why prefixes, not markers.** The alternative was bracketing `specfill`'s region
with directive comments (`//spec:doc` … `//spec:end`). Those are invisible in
godoc and safe by default — no marker, nothing touched — but they add lines to
every declaration and say nothing about *why* text is outside spec. Prefixes
reuse the shape the notes already have, work the same in generated and
hand-written files, and force each piece of non-spec text to say what kind of
note it is. The cost is that unmarked text defaults to spec-owned, which is why
every rewrite is reviewed as a diff, and why generators, once they emit no API
prose of their own, may not emit spec-owned text at all (see `specfill`).
## 1.7 Errors versus tolerated gaps
At CL 831485, of the 782 `(receiver, name)` pairs declared for more than one
target, not one had divergent parameter or result *types* — but 72 disagreed on
parameter names and 16 on result names. So spec is *preventive, not remedial*:
there is no backlog of type-level fixes, and incremental migration is not racing
decay.
- **Signature mismatch is an error.** For an operation spec knows about, a
disagreement is a bug in spec or in a generator.
- **A missing spec function is tolerated** until the transition completes.
- **Parameter and result names are checked** where the declaration names them
(unnamed ones are filled from spec; §1.8), for two reasons: they appear in
documentation and so must be consistent with the documentation's source, and
several past bugs have come from confused or swapped argument names.
- **`specfill` considers exported declarations only.** This is what makes the
tolerated-miss list converge: all three `ops_internal_*.go` files (amd64,
arm64, sve) contain zero exported declarations, so every compiler-only
operation falls outside `specfill`'s view without needing a category. The
permanent residue is small and enumerable: the `String`, `Len`, `ToArch`
methods and `xFromArch` functions.
## 1.8 Named results
Spec names its results and its docs reference those names (`z[i] = x[i] +
y[i]`). Almost nothing in the shipped API names its results (SPEC-STATS.txt §2).
When spec's doc becomes archsimd's doc, the references to `z` in the docs won't
make sense unless we carry over spec's result names as well.
Decision: **where spec covers a declaration, its parameter and result names
come from spec**, so they agree with the doc that also comes from spec. Naming
every result is *not* a goal: a name is worth having where a doc or a body uses
it, and naming the rest blindly serves no one. Declarations spec does not cover
keep whatever names they have until `spec-all` covers them.
`specfill` enforces this without touching bodies. It rewrites unnamed
arguments/results in the API (including `_`) to match spec, and reports a
mismatch error if both sides name an argument/result and the names disagree. For
a bodyless declaration, all that matters is that the names agree with the doc. A
body can refer to named arguments but not unnamed ones, so filling unnamed ones
is safe, while named ones must be checked. SPEC-STATS.txt §2 counts the two
cases separately, the same way.
## 1.9 Behavioral conformance is a parallel track
Behavioral conformance has **no hard dependency on the API documentation
track**. Both consume the same input — spec coverage — independently, and they
catch disjoint bug classes: a swapped argument in a new architecture's `Sub` has
an identical signature and identical docs, and only conformance sees it.
The order for bootstrapping conformance matters. **Validate spec against amd64
first** — it is the most mature target, and every failure is a spec bug unless
proven otherwise. There may be bugs or hardware-specific corner cases in the
amd64 implementation, but bugs in spec are far more likely. Once spec and amd64
agree, **then flip**: spec becomes the oracle and gates arm64, wasm, SVE, and
any architecture added later. Getting this backwards means conformance failures
against correct implementations, and the obvious move is then to "fix" the
implementation.
Harness requirements implied by the bug classes we actually see: per-type
edge-case inputs (0, ±1, MinInt, MaxInt, ±0, ±Inf, NaN, saturation boundaries)
crossed with random vectors — uniform random will not reliably hit saturation or
NaN corners; and explicit mask patterns, since an inverted mask sense passes
random testing about half the time.
**This is a path we have already gone down successfully.**
`simd/internal/test_helpers` carries per-type input vectors — `float32s` through
`uint8s`, plus `bools` for masks — built by cycling a hand-picked list of
exactly those corners (±0, ±Inf, quiet and signalling NaN, `MaxFloat`, and the
8- and 16-bit saturation edges) up to a fixed length. In
`archsimd/internal/simd_test/helpers_test.go`, the
`forSlice`/`forSlicePair`/`forSliceTriple`/`forSlicePairMasked` helpers walk
windows over them for the cross-product, `map1`/`map2`/`map3`/`mapCompare` build
the expected result elementwise — the same shape a spec body has — and
`grouped2` already handles the 128-bit grouping that gives `doc-explore`
trouble. `checkSlicesLogInput` even takes a tolerance, which is a first cut at
the vocabulary `impl-defined` has to settle.
`conform-fixed` need not reuse that code literally, but it should not start over
either: what it adds is generating the cases from spec rather than hand-writing
them per operation.
## 1.10 Progress is measured by name, not by normalization
`apisum` needs normalization because its job is to *test* whether names mean one
thing. Once we enforce that they do, **the method name is the equivalence class**,
exactly and for free: coverage is a set intersection, and `cmd/specstats` already
computes it. No name normalization, and no complex pattern matching rules to
maintain or throw away.
Caveat, worth knowing rather than fixing: templated-name families like
`LoadInt8x16`, `LoadInt8x32`, etc, that all come from one spec function inflate
the denominator of any coverage metrics. That is arguably correct for measuring
API *surface*. Reporting both numbers — "spec has N functions covering M names"
— is more informative than either alone.
---
# Part 2 — Task graph
The 27 tasks fall into six **tracks**. Tracks are a way to divide the work among
people and to make Part 3 navigable. They are not phases and don't imply any
ordering on tasks. Tasks in each track generally cluster in the task dependency
graph, but there are inter-track dependencies.
| | Track | What it is | Tasks |
|---|---|---|---|
| **A** | Decisions | Human judgment calls; no prerequisites, all block downstream work | `width`, `compiler-ops`, `convert-lo`, `mask-bits`, `impl-defined` |
| **B** | Instrumentation and enforcement | Build `specfill`, then make it authoritative | `specfill`, `fill-gen`, `named-results`, `gen-docs`, `fill-enforce` |
| **C** | Conformance | Behavioral testing against `simdref` | `ci-audit`, `conform-fixed`, `validate-amd64`, `conform-scalable`, `conform-flip` |
| **D** | Spec content | Writing spec functions, docs, and bodies — the shared bottleneck | `doc-explore`, `spec-common`, `spec-all` |
| **E** | Retiring the old inputs | The five non-spec doc sources, then `categories.yaml` | `comments-yaml`, `doc-triage`, `handwritten-fill`, `template-docs`, `cat-rehome`, `compiler-ops-impl`, `cat-delete` |
| **F** | simdgen internals | The signature refactor, then sourcing signatures from spec | `sig-refactor`, `sig-from-spec` |
## Dependencies
A task is startable when every prerequisite is ticked. Hard dependencies are
listed as `Needs`; soft dependencies — work is possible but wasteful or
lower-value without them — as `Prefers`.
Tasks are named, not numbered, so that a task's identity survives inserting
another one next to it, and so that a CL saying "implements `cat-rehome`" still
means something a year later. Ordering lives in the row order instead: the table
is topologically sorted (both `Needs` and `Prefers` count as edges), with ties
broken by track, so reading it top to bottom is a legal order to do the work in,
and `cmd/spectasks` fails if an edit breaks that. Add a task by putting its row
wherever its prerequisites and track allow.
| ✓ | T | Task | What it is | Needs | Prefers |
|---|---|---|---|---|---|
| [x] | A | `width` | Resolve scalable width | — | — |
| [ ] | A | `compiler-ops` | Compiler-only ops decision | — | — |
| [ ] | A | `convert-lo` | `ExtendLo*`/`ConvertLo*` shape and naming | — | `width` |
| [ ] | A | `mask-bits` | Mask bitmap load/store shape and naming | — | `width` |
| [ ] | A | `impl-defined` | Implementation-defined behavior policy | — | — |
| [x] | B | `specfill` | API walker + report mode | — | — |
| [x] | B | `fill-gen` | Wire `specfill` into the generators | `specfill` | — |
| [x] | B | `named-results` | Reconcile parameter and result names with spec | `fill-gen` | — |
| [x] | B | `gen-docs` | Inject spec docs into generated output | `fill-gen` | `named-results` |
| [x] | B | `fill-enforce` | Signature + name mismatch becomes an error | `named-results` | — |
| [x] | C | `ci-audit` | CI/hardware audit | — | — |
| [ ] | C | `conform-fixed` | Conformance harness, fixed-width types | `ci-audit` | `specfill` |
| [ ] | C | `conform-scalable` | Conformance harness, scalable types | `width`, `conform-fixed` | — |
| [x] | D | `doc-explore` | Doc-mechanism exploration on a hard family | — | — |
| [x] | D | `spec-common` | Spec the common operations | `width`, `doc-explore` | `specfill` |
| [ ] | C | `validate-amd64` | Validate spec bodies against amd64 | `conform-fixed`, `spec-common`, `impl-defined` | — |
| [ ] | C | `conform-flip` | Flip conformance direction | `validate-amd64`, `conform-scalable` | — |
| [ ] | D | `spec-all` | Bulk spec migration | `spec-common`, `convert-lo`, `mask-bits` | `specfill`, `fill-enforce` |
| [ ] | E | `doc-triage` | Hand-written doc conflict triage | `specfill` | `spec-common` |
| [x] | E | `comments-yaml` | Drop spec-covered docs from `midway/comments.yaml` | `gen-docs`, `spec-common` | `fill-enforce` |
| [ ] | E | `handwritten-fill` | Enable `-w` on hand-written files | `gen-docs`, `doc-triage` | — |
| [ ] | E | `template-docs` | Retire generator-supplied API docs | `gen-docs`, `spec-all` | — |
| [ ] | E | `cat-rehome` | Re-home `categories.yaml` non-doc fields | `template-docs` | — |
| [ ] | E | `compiler-ops-impl` | Implement compiler-only ops representation | `compiler-ops`, `cat-rehome` | — |
| [ ] | F | `sig-refactor` | simdgen signature refactor | — | — |
| [ ] | F | `sig-from-spec` | simdgen sources signatures from spec | `sig-refactor`, `cat-rehome`, `compiler-ops-impl` | — |
| [ ] | E | `cat-delete` | Delete `categories.yaml` | `sig-from-spec` | — |
**Critical path.** [SPEC-TASKS.dot](SPEC-TASKS.dot) draws it in red: every task
on some longest chain of `Needs` edges. Slipping any of them slips the whole
project. `cmd/spectasks` derives it from the table, so it is not restated here.
Track B has no prerequisites, so it can start immediately and run to completion
at its own pace — though `gen-docs` gates three of track E's tasks.
`sig-refactor` is independent of everything until `sig-from-spec`.
## Milestones
Milestones carry no work of their own; they are the points worth announcing. A
milestone is reached when its tasks are ticked, so there is no separate state to
maintain and nothing here to keep in sync.
| Milestone | Reached after | What it means |
|---|---|---|
| **M1 — Enforced** | `fill-enforce` | Spec is authoritative for what it covers: a name or signature that drifts from it fails the build, in both packages, on all four targets. |
| **M2 — Documented** | `comments-yaml`, `handwritten-fill`, `spec-all`, `template-docs` | Every operation's doc comment in both public packages derives from spec — SPEC-STATS.txt's "docs sourced from spec" reads 100%. |
| **M3 — Gated** | `conform-flip` | Conformance runs spec-first, so an architecture added later must match spec before it can land. |
| **M4 — Complete** | `cat-delete` | `categories.yaml` is gone and spec is the only hand-written description of the API; see the section below. |
## The graph
[SPEC-TASKS.dot](SPEC-TASKS.dot) draws the two tables above; the rendered graph
carries its own key. Track is node colour rather than a cluster, because a
cluster has to be laid out contiguously and that fights the left-to-right
dependency order the graph exists to show.
dot -Tsvg SPEC-TASKS.dot -o SPEC-TASKS.svg
It is generated, and its generator is also a checker. This document states its
dependencies three times — Part 2's table, Part 3's `Needs:`/`Prefers:` lines,
and Part 3's `Blocks:` lines, which are the same edges reversed — so
`go test ./cmd/spectasks` fails if they disagree, or if the graph has not been
regenerated since. It also derives the critical path.
## Start here
The startable tasks are the ones with no unticked `Needs` in the dependency
table. For a team of one to three, this is a reasonable opening allocation,
covering every task with no `Needs` at all:
- **Strongest on spec's type model:** `width`, then `convert-lo` and
`mask-bits`. These root the longest chain and cost the most if deferred.
- **Owns API semantics:** `impl-defined` and `compiler-ops` — pure
decisions, no code, and everything downstream waits on them.
- **Likes tooling:** `specfill`, then `fill-gen` and `named-results`.
- **Working alone on something large:** `sig-refactor`. No prerequisites,
verifiable byte-for-byte at every step.
- **Cheap and worth doing early regardless:** `doc-explore` and `ci-audit`.
## What "done" looks like
After `cat-delete`, `simd/internal/spec` is the only hand-written description of
the API: every operation's name, signature, and documentation derives from it,
in both public packages and on all four targets, and `specfill` fails the build
if any of them drift. `categories.yaml` is gone. `midway/comments.yaml`
documents only what spec does not describe: the vector types, `VectorSize`,
`String` and `Len`. wasmgen and tmplgen no longer carry doc strings.
Hand-written implementations remain hand-written, but their operation doc
comments are generated.
simdgen still exists and is still unification-based — §1.1 explains why. What
changes is its job: the unifier no longer defines the API, it only expands one
`go_GOARCH.yaml` entry into all the concrete machine instructions matching it.
`go_GOARCH.yaml` stays, and so do its quirk fields, because instruction
irregularity is real and has to live somewhere. The compiler backend generation
is untouched throughout.
---
# Part 3 — Task details
Many completion criteria are "generated output is byte-identical", checked with
`go run . -diff` from `_gen`. That runs every generator, including simdgen for
all three of its targets, so it needs the XED and ARM ISA data first — see
`fetch-xed.sh` and `fetch-arm64.sh`. There is no `-arch` flag on the driver;
`-arch` belongs to `simdgen` when run directly.
## Track A — Decisions
Human judgment calls. None has prerequisites, and each blocks downstream work —
which is why they lead. Each is small in code and large in consequence.
### `width` — Resolve scalable width
**Done:** [x] · **Needs:** — · **Blocks:** `spec-common`, `conform-scalable` · **Risk:** `risk-concrete-width`
`simd/internal/spec/types.go` hardcodes `scalableWidth = 4096`, and `simdref` is
built at that width, while `archsimd`'s SVE uses the runtime `vl()`. A
conformance test of scalable types therefore compares vectors of different
lengths. Spec's own TODO flags this and it is unresolved.
4096 bits in particular causes trouble for operations that index into vectors,
such as shuffles, which is why every platform other than RISC-V RVV is 2048 bits
or less. Resolving this task may require deciding whether we'll ever support
vectors more than 2048 bits wide.
Note what is *not* wrong: spec's type system was designed for scalable widths and
handles them. The gap appears only when spec is used as a concrete conformance
oracle, and the open question is narrow — whether spec can execute at a
caller-chosen width without disturbing the generic type machinery.
This is first because the fix may change spec's type model — making width a
runtime parameter rather than a compile-time constant — which reaches every spec
function body. Resolving it after a hundred bodies exist is far worse than
resolving it now, while there are few (SPEC-STATS.txt §1).
**Commits.** (1) Write up the options with their consequences for `Vec`,
`makeVec`, `lanes`, `refgen`, and the conformance harness; seen so far:
parameterize width at spec execution, make `simdref` width-configurable at
construction, or scope conformance to fixed widths initially and defer. (2)
Implement the chosen approach in `internal/spec`. (3) Regenerate
`internal/simdref` and confirm `cmd/specls` output is unchanged, or explain what
changed.
**Done when** a scalable-typed operation can be executed through `simdref` at a
width matching a chosen target, and the choice is documented in
`internal/spec/doc.go`.
### `compiler-ops` — Compiler-only ops decision
**Done:** [ ] · **Needs:** — · **Blocks:** `compiler-ops-impl`
Decide how the compiler-only entries — those that produce no exported
declaration (SPEC-STATS.txt §4) — and the `noTypes` / `noGenericOps` switches
are represented once `categories.yaml` is gone. This is the only hard blocker in
the chain without an obvious answer, so make it early.
Note §1.7: because `specfill` is scoped to exported declarations, this does
*not* block the checker or conformance tracks.
The design note must also say where the following go once `categories.yaml` is
gone: the compiler-only entries' `documentation:`, which feeds the doc comments
of unexported methods in `archsimd/ops_internal_*.go`; `commutative`, `signed`
and `out` on those entries; and the `go_arm64.yaml` use of `noGenericOps`. This
documentation is strictly internal-facing, so it is held to a lower standard
than user-facing API docs. It need not come from spec, go through `specfill`, or
satisfy M2. Wherever it lands is acceptable as long as it is kept next to the
compiler-only op's other description.
**Commits.** One design note in this document or in `internal/spec/doc.go`,
listing all compiler-only operations from `_gen/simdgen` (`grep -h "go: [a-z]"
ops/*/categories.yaml | sort -u`; also the capitalized `noTypes` entries) and
stating for each family where its description, documentation, and remaining
fields live once `categories.yaml` is gone. Candidates: a `//specgen:unexported`
directive; a fourth simdgen input holding compiler-only ops; or leaving them in
`go_GOARCH.yaml` on the grounds that they are per-architecture lowering details
rather than API.
**Done when** the decision is written down, including documentation and all
remaining fields, and `compiler-ops-impl` could be implemented from it without
further design work.
### `convert-lo` — `ExtendLo*` / `ConvertLo*` shape and naming
**Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects`
Settle how the low-elements widening family is shaped and named, across both
fixed-width and scalable vectors. An API design decision, not a mechanical fix.
Three entangled problems:
1. **Shape.** `ExtendLoLToZ` declares `x Vec[E,W]` and `z Vec[zE,W]` — the same
total width. The shipped API widens instead: `Int8x16.ExtendLo8ToInt64()`
takes 128 bits and returns 512. Spec therefore misses 8 real declarations and
invents width-preserving ones no platform has.
2. **Naming.** `//specgen:name ExtendLo{zL}To{zE}` embeds the *result lane
count*, so the same conceptual operation gets a different name at each width.
The API has settled on 128-bit receivers only, which hides this.
3. **Scalable.** The `converts.go` TODO already notes the literal `zL` cannot
work for scalable vectors, and suggests dropping the number entirely since it
is implied by the receiver and target types — e.g. `Uint8x16.ExtendLo2ToUint64`
is really the low *eighth*, not the low half, so the number is arguably
misleading even where it works.
Any renaming here is API-visible and should be decided before `spec-all`
migrates the Converts family, which is where the bulk of `categories.yaml`'s
Converts documentation lands.
**Commits.** (1) Write up the options and their consequences for fixed-width and
scalable vectors, including whether the lane count stays in the name. (2)
Re-shape `ExtendLoLToZ`, and re-check `ConvertLoLToZ`, whose `zL<xL` constraint
currently admits conversions no hardware offers. (3) If the API name changes,
land the rename through the generators as its own CL, separate from the spec
change.
**Done when** `cmd/specls` produces exactly the `ExtendLo*` declarations the API
has, plus whatever maximalist extras are deliberate, and the scalable naming
question is answered in `internal/spec/doc.go`.
### `mask-bits` — Mask bitmap load/store shape and naming
**Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects`
Settle the scalable counterpart of fixed-width `ToBits` and `MaskNxMFromBits`
(e.g. `Mask8x16FromBits`), which today are `Mask*s.Store` and `LoadMask*s`. Also
settle what `Store` and `ToBits` mean on portable `simd.Mask*s`, whose midway
docs currently describe element semantics.
The API declares `Store(bits []uint16)` only on SVE's scalable masks
(`archsimd/types_sve.go`), paired with `LoadMask8s(bits []uint16)` etc. It
stores SVE predicate bits at byte granularity. No fixed-width mask has `Store`;
fixed-width masks have `ToBits() UintN` instead (spec: `MaskToBits`,
`internal/spec/masks.go`, constrained to `FixedWidth`). So `Mask*s.Store` has a
different meaning from every other `Store` — a candidate violation of the "one
name, one meaning" rule. Its bit layout is SVE's representation, not a per-lane
one, which arguably fails §1.3's "claim about the operation's own semantics"
test.
**Commits.** (1) Write up the options and their consequences for fixed-width and
scalable masks, and for the portable package. (2) Change spec to match. (3) If
the API name changes, land the rename through the generators as its own CL.
**Done when** SPEC-STATS.txt §5 lists no mask `Store` defect, and the choice is
recorded in `internal/spec/doc.go`.
### `impl-defined` — Implementation-defined behavior policy
**Done:** [ ] · **Needs:** — · **Blocks:** `validate-amd64` · **Risk:** `risk-impl-defined-behavior`
Decide what counts as implementation-defined behavior in the SIMD API, and how
spec expresses it so conformance testing stays meaningful. Architectures
legitimately differ on corners — the last bits of a float result, out-of-range
shuffle indices, saturation edges, NaN propagation and signed zero. Spec's
`math.go` already carries a TODO on `DotProductPairs` overflow that names the
problem and floats an "architecture dependent" designated panic as one possible
answer.
Two halves, and both need answering:
1. **What is implementation-defined?** A per-operation API-semantics judgment.
Costly in both directions: declare too much and conformance stops catching
real bugs; declare too little and correct implementations fail against spec.
2. **How is it written down?** There are at least two major categories of
implementation-defined behavior: exceptional-value cases (e.g., overflow
behavior is fully undefined) and precision (all architectures roughly agree,
but maybe not to the last digit). Spec bodies return exact values today, and
there is no vocabulary for "any value here", "within N ulp", or "unspecified
but deterministic per platform". Whatever is chosen must be understood by
both `refgen` and the conformance harness, and it changes what a spec body
looks like — which is why this wants settling before the bulk of them are
written. The existing suite's `checkSlicesLogInput` already takes a per-call
tolerance (§1.9), which is prior art for the numeric half; it says nothing
about the "unspecified but deterministic per platform" half.
**Commits.** (1) Survey the known cases — `DotProductPairs` overflow, float
conversion rounding, shuffle index out of range, saturation edges, NaN/±0
handling — classifying each as specified, implementation-defined, or undefined.
(2) Choose and document the spec vocabulary in `internal/spec/doc.go`. (3)
Implement it far enough that one operation with a genuinely
implementation-defined corner round-trips through `refgen` and the harness.
**Done when** an operation with an architecture-dependent corner can be written
in spec, generated into `simdref`, and conformance-tested without a false failure
on any supported target.
## Track B — Instrumentation and enforcement
Build `specfill`, then make it authoritative. Independent of tracks C, D and F;
`gen-docs` gates three of track E.
### `specfill` — API walker + report mode
**Done:** [x] · **Needs:** — · **Blocks:** `fill-gen`, `doc-triage`
The instrument the rest of the plan reads from.
`specfill` builds the whole of `specdoc.Fill` and `cmd/specfill` — the walk, the
spec index, the comparison, and the rewrite of doc comments and parameter/result
names. `fill-gen` wires it into generators with rewriting off; `named-results`
and `gen-docs` then turn on name filling and doc injection there, and
`handwritten-fill` turns `-w` on for hand-written files.
**Interfaces.** `specdoc.Fill` is the library entry point — Go source in, Go
source out: `func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`.
Generators call it on their buffer before writing (§1.5). When verification
checks fail, `Fill` returns a `*Report` as an error containing the findings
enumerated below. Callers can recognize this error via `errors.As` if they
choose to inspect or combine findings across files. Option fields:
- **RejectUnknown** controls whether an exported declaration with no spec entry
is an error (recorded in `Report.UnknownDecls`). Off during the transition;
generators flip it on after `spec-all`, so that adding an operation without
writing its spec function fails at generation time rather than silently escaping
coverage.
- **AllowDocRewrite** controls whether a declaration spec covers may already
carry spec-owned paragraphs (§1.6). When false (default), each one that does
is an error (recorded in `Report.UnexpectedDocs`); implementation notes and
directives are always allowed. Once generators emit no API prose of their own
(`template-docs`), this is their guard against silent doc loss
(`risk-silent-doc-loss`): a generator paragraph missing its note prefix is an
error rather than something `Fill` quietly replaces. Until then generators set
it to true. Hand-written files carry prose by design, so `cmd/specfill` always
sets it to true and relies on `-diff` and its `-w` guards instead.
- **AllowNameMismatches** controls whether name mismatches are tolerated. When
false (default), they are errors. Generators can set it to true during the
transition, until `named-results` reconciles them.
Two more fields select what `Fill` rewrites, for callers that want only part of
it: **NoFillDoc** leaves doc comments alone, and **NoFillNames** leaves unnamed
arguments unnamed (and so reports them as name mismatches). By default `Fill`
does both. Finally, the **Filename** field labels positions in findings.
**Names.** `Fill` fills and checks names as §1.8 describes: an argument the
declaration leaves unnamed (or names `_`) takes spec's name, and a named
argument whose name differs from spec's is a mismatch. It rewrites a
declaration's names only when that declaration has no type or name mismatch, so
a mismatch has to be fixed before the rest of its declaration's names are
filled.
`cmd/specfill` is a `gentools`-based command that applies `specdoc` to
hand-written files: it walks the non-test `.go` files in `simd` and `archsimd`,
skips generated ones (their generators call `Fill` themselves), calls `Fill` on
each with `AllowDocRewrite`, and prints the combined report. `-diff` shows what
a rewrite would change. With `-w`, rewrite in place; `-w` aborts without
rewriting when the report has any findings, or if there are uncommitted changes,
so its git diff is the complete story of what it did. The `-no-fill-doc` and
`-no-fill-names` flags set the corresponding options. CI enforcement is a `go
test` in this package — `fill-enforce` wires it — not a command invocation.
**Report.** The report is the permanent enforcement layer — every
item in it remains useful after the transition as the API continues to evolve,
and is what `fill-enforce` runs in CI. Coverage and name relations are
transition metrics that converge to their targets; they stay in
`cmd/specstats` / SPEC-STATS.txt and are not part of this report.
The report holds only errors; it does not preview rewrites. Every consumer of
`specdoc` is a `gentools` program, `cmd/specfill` included, so `-diff` already
shows exactly what a rewrite would change against the files on disk, including
any text it would drop. That diff is how rewrites are reviewed, and a report that
recomputed it would only be a second copy to keep in sync.
The possible errors are:
1. **Type signature mismatches** — parameter or result types, or their
counts, disagree between spec and the declaration the walker found.
2. **Name signature mismatches** — a named parameter or result whose name
differs from spec's. Unnamed ones are filled rather than reported, unless
`NoFillNames` is set. Omitted when `AllowNameMismatches` is set.
3. **Check (a) violations** — §1.6 ordering violations in declarations spec
covers. `Fill` leaves such a comment as it is. Declarations spec does not
cover are not checked: the ordering rule exists so rewrites are idempotent,
and `Fill` never rewrites them. Hand-written files in particular may hold
declarations spec will never cover. Each declaration is checked from the
moment spec covers it.
4. **Unknown declarations** — exported declarations with no spec entry, when
`RejectUnknown` is set.
5. **Unexpected doc comments** — declarations spec covers that carry
spec-owned paragraphs, when `AllowDocRewrite` is false.
**Commits.** (1) `specgen`: add an index over `Load()`'s result keyed by `(recv,
name)`; add a function that classifies comment paragraphs as spec-owned or
implementation note per §1.6; reject spec docs that break check (b). (2) New
package `_gen/specdoc` and new command `_gen/cmd/specfill`: AST declaration
walker, signature and comment ordering comparison, report findings implementing
the `error` interface, and `AllowDocRewrite` / `AllowNameMismatches` options.
Signature comparison translates the parsed `*ast.FuncDecl` into spec's own
representation — its types into `specexpr` types, which suffices because no
qualified types exist (§1.4) — and compares that structurally against the
`specgen.Func`. (3) Rewrite engine in `specdoc.Fill` (both doc comments and
signature names) and in-place rewriting (`-w`, `-no-fill-doc`, `-no-fill-names`)
in `cmd/specfill`.
**Done when** `specfill`, run on the current tree, writes nothing (without `-w`)
and prints all report findings in position order.
`SPEC-STATS.txt` is the sanity check for signatures: `cmd/specstats` computes
the same type and name mismatches, counting names the way `Fill` does (§1.8), so
`specfill`'s report should agree with it line for line on those. Treat any
disagreement as something to explain before shipping. Coverage and name
relations stay in `specstats` as transition metrics; once they reach their
targets, `specstats` can be retired.
### `fill-gen` — Wire `specfill` into the generators
**Done:** [x] · **Needs:** `specfill` · **Blocks:** `named-results`, `gen-docs`
Each generator passes its buffer through `specfill` before writing, with all
rewriting off for now: `NoFillNames` (`named-results` turns name filling on) and
`NoFillDoc` (`gen-docs` turns doc injection on). Generators also set
`AllowNameMismatches`, because names still disagree with spec until
`named-results`, and `AllowDocRewrite`, because they still emit API prose of
their own until `template-docs`.
So this task only wires the generators up, and generated output does not
change. What does change is that `Fill`'s remaining checks now run at
generation time: a type mismatch or a check (a) violation in generated output
fails generation. Neither can be tolerated by an option, and as of `specfill`'s
completion there are none in generated output.
**Commits.** (1) Add a narrow post-processing hook to `gentools` that does not
make gentools depend on specgen (see §1.5). (2) Wire simdgen. (3) Wire tmplgen,
wasmgen, midway.
**Done when** all four generators route through `specfill`, and generated
output is byte-identical to the previous tree.
### `named-results` — Reconcile parameter and result names with spec
**Done:** [x] · **Needs:** `fill-gen` · **Blocks:** `fill-enforce`
Every declaration spec covers carries spec's parameter and result names. See
§1.8. Declarations spec does not cover are out of scope; they gain names when
`spec-all` covers them.
SPEC-STATS.txt §2 has the counts, split the way `specfill` treats them:
- **Named arguments whose names disagree with spec**, in generated or
hand-written code. Each is fixed in whichever of spec or the API has the worse
name — spec, a generator's template or input, or a hand-written file; either
way the diff touches only names, and doc text that mentions them. These come
first: `Fill` fills no names in a declaration with a mismatch.
- **Unnamed arguments**, which `Fill` fills from spec. In generated code, that
means turning name filling on in the generators. In hand-written files,
`specfill -w -no-fill-doc` fills them; `-w` refuses while the report has
findings, so this also needs any check (a) violations in hand-written files
cleared (they are also `doc-triage`'s).
Cross-target parameter-name divergence (also SPEC-STATS.txt §2) is only fixed
here for pairs spec covers. The rest reach zero as `spec-all` covers them, when
each new spec function has to agree with every target at once.
**Commits.** (1) Reconcile name mismatches, grouped by generator or operation
family, each commit with its regenerated output. (2) Stop setting `NoFillNames`
and `AllowNameMismatches` in the generators, with the regenerated output: name
filling is on, and a name mismatch in generated output fails generation. (3)
Fill unnamed arguments in hand-written files with `specfill -w -no-fill-doc`.
**Done when** SPEC-STATS.txt §2's name mismatches and unnamed counts are zero,
and the generators run without `NoFillNames` or `AllowNameMismatches`.
### `gen-docs` — Inject spec docs into generated output
**Done:** [x] · **Needs:** `fill-gen` · **Prefers:** `named-results` · **Blocks:** `comments-yaml`, `handwritten-fill`, `template-docs` · **Risk:** `risk-silent-doc-loss`
Turn on doc injection in the generators by dropping `NoFillDoc`. Generators keep
setting `AllowDocRewrite` until `template-docs` removes their own prose.
Prefer landing this after `named-results`: spec's docs refer to argument names
(§1.8), so injecting them before the generators fill those names leaves docs
that mention names the signature does not bind.
Expect a real diff, not a no-op: spec already overlaps several hundred
`(receiver, name)` pairs (SPEC-STATS.txt §1), and almost none of the existing
doc text matches spec's (SPEC-STATS.txt §3). So this rewrites that many doc
comments. Use `go run . -diff` first to size the change exactly.
Before any generator's docs are injected, its implementation notes must follow
§1.6, or the first rewrite silently drops them. Known gaps: some notes say `CPU
Features:` or omit a colon; simdgen's performance caveats ("A non-constant value
of shift may result in significantly worse performance", and the `SetElem`
jump-table line) have no prefix, and the `SetElem` one sits directly above its
`Asm:` line with no blank line between them. `specfill`'s check (a) violations
and a `-diff` of the output with injection on are the list to work from.
**Commits.** (1) Fix note prefixes in simdgen, tmplgen, wasmgen and midway
templates — `Performance:` for the caveats, a blank line before every note —
with the regenerated output; the diff should touch only note lines. (2) Turn on
doc injection in simdgen. (3) Turn it on in tmplgen, wasmgen, midway. Commits
(2) and (3) include their regenerated output; verify that output changes only
for operations spec covers, and review every changed doc comment to make sure
nothing was unintentionally dropped. (4) If `fill-enforce` is already done, drop
`NoFillDoc: true` from `cmd/specfill.TestFill` for generated files to start
enforcing documentation matches.
**Done when** generated output has no check (a) violations, all four generators
inject docs, nothing was dropped without review, and the only differences
against the previous tree are doc comments for operations spec covers.
### `fill-enforce` — Signature + name mismatch becomes an error
**Done:** [x] · **Needs:** `named-results`
For operations spec covers, disagreement fails. Missing spec functions remain
tolerated. Note this does **not** depend on `sig-refactor`: `specfill` compares
emitted source, not simdgen's internal model.
Generated output is already checked as it is generated: type mismatches fail
generation from `fill-gen` on, and name mismatches from `named-results` on.
That misses hand-written files, and generated files edited by hand without
rerunning their generator. This task closes both with a `go test` in
`cmd/specfill` that runs `Fill` over every non-test `.go` file in `simd` and
`archsimd`, generated or not, and fails on any finding. It sets
`AllowDocRewrite`, since those files carry spec-owned docs by design, whether
injected or hand-written.
**Commits.** One, adding the test.
**Done when** a deliberately introduced signature or argument-name change in any
generated or hand-written exported declaration fails the build.
## Track C — Conformance
Behavioral testing against `simdref`. Parallel to everything else; shares only
the spec-coverage bottleneck.
### `ci-audit` — CI/hardware audit
**Done:** [x] · **Needs:** — · **Blocks:** `conform-fixed` · **Risk:** `risk-conformance-env`
Determine what can actually execute tests for amd64, arm64, wasm, and SVE —
natively or emulated. The conformance suite's value for other architectures
depends on it.
"Execute tests" means: build `simd` and `archsimd` for the target and run
`go test` against real or emulated hardware, at a speed that tolerates a few
thousand generated conformance cases. SVE needs SVE-capable hardware or QEMU, and
wasm needs a runtime.
**Done when** one document or issue records, per target, whether it can run
today, on what, how fast, and what it would take if not — marking each runnable,
runnable-with-work (with the work named), or not runnable, so `conform-fixed`
knows which targets it is writing tests for.
**Results**
| amd64 | gotip-linux-amd64 | gotip-linux-amd64_avx512 |
| ---------------- | ----------------- | ------------------------ |
| AVX | ✔ | ✔ |
| AVX2 | ✔ | ✔ |
| AVX512 | ✘ | ✔ |
| AVX512BITALG | ✘ | ✔ |
| AVX512GFNI | ✘ | ✔ |
| AVX512VAES | ✘ | ✔ |
| AVX512VBMI | ✘ | ✔ |
| AVX512VBMI2 | ✘ | ✔ |
| AVX512VNNI | ✘ | ✔ |
| AVX512VPCLMULQDQ | ✘ | ✔ |
| AVX512VPOPCNTDQ | ✘ | ✔ |
| AVXAES | ✔ | ✔ |
| AVXPCLMULQDQ | ✔ | ✔ |
| AVXVNNI | ✘ | ✔ |
| FMA | ✔ | ✔ |
| SHA | ✘ | ✔ |
| VAES | ✘ | ✔ |
| VPCLMULQDQ | ✘ | ✔ |
For amd64, use the gotip-linux-amd64_avx512 gomote for testing.
| arm64 | gotip-linux-arm64 | qemu-aarch64 -cpu neoverse-n2 | qemu-aarch64 -cpu neoverse-v1 |
| --------------- | ----------------- | ----------------------------- | ----------------------------- |
| PMULL | ✔ | ✔ | ✔ |
| SVE | ✘ | ✔ | ✔ |
| SVE2 | ✘ | ✔ | ✔ |
| SVE vector size | n/a | 128 | 256 |
`qemu-aarch64 -cpu neoverse-n1` (which matches the capabilities of
gotip-linux-arm64) is roughly 7x slower than `gotip-linux-arm64`, so well within
the range of practical conformance testing.
For NEON, use the gotip-linux-arm64 gomote for testing. For SVE, use
`qemu-aarch64 -cpu neoverse-v1`.
| wasm | js-node | wasip1-wazero |
| ------- | ------- | ------------- |
| SIMD128 | ✔ | ✔ |
`node` is about 3x faster than `wazero`.
For wasm, use `node` for testing.
### `conform-fixed` — Conformance harness, fixed-width types
**Done:** [ ] · **Needs:** `ci-audit` · **Blocks:** `validate-amd64`, `conform-scalable` · **Prefers:** `specfill`
Generated tests comparing `archsimd` against `internal/simdref`. The generator's
input is spec (hand-written source), so this is not a generator reading another
generator's output.
**Commits.** (1) Input generation: start from
`archsimd/internal/simd_test/helpers_test.go`, which already has the per-type
edge-case vectors, the mask patterns and the window-walking helpers (§1.9);
decide whether to lift them into a package the generated tests can import. (2)
Test generator producing one test per `(recv, name)` spec covers. (3) Wire into
the SIMD test suite, skipping operations whose spec body panics.
**Done when** every operation spec covers, on every target `ci-audit` marked
runnable, is exercised against `simdref` with both edge-case and random inputs,
and failures name the operation, the input, and both results.
### `conform-scalable` — Conformance harness, scalable types
**Done:** [ ] · **Needs:** `width`, `conform-fixed` · **Blocks:** `conform-flip`
Extend `conform-fixed`'s harness to the scalable types (`Int8s` … `Float64s`,
`Mask8s` … `Mask64s`) on SVE. Separate from `conform-fixed` because it cannot be
written until `width` settles how spec executes at a concrete width: the harness
must construct `simdref` values at the width the hardware reports through `vl()`
rather than at spec's fixed `scalableWidth`.
**Commits.** (1) Teach input generation to size vectors from the runtime vector
length. (2) Extend the test generator to scalable receivers. (3) Wire into the
suite, gated on SVE availability per `ci-audit`.
**Done when** scalable operations spec covers are conformance-tested on SVE at
the hardware's actual vector length.
### `validate-amd64` — Validate spec bodies against amd64
**Done:** [ ] · **Needs:** `conform-fixed`, `spec-common`, `impl-defined` · **Blocks:** `conform-flip`
Run conformance with amd64 as the oracle. Every failure is a spec bug until
proven otherwise. See §1.9.
**Commits.** One per batch of spec fixes; keep spec changes separate from harness
changes so the reason for each is legible.
**Done when** the harness passes on amd64 for every operation spec covers, with
every remaining difference either fixed in spec or recorded as
implementation-defined under `impl-defined`'s policy.
### `conform-flip` — Flip conformance direction
**Done:** [ ] · **Needs:** `validate-amd64`, `conform-scalable`
Spec becomes the oracle for every target, not just amd64. The second half is the
contributor-facing note — likely in `internal/spec/doc.go` and the SIMD README —
describing the workflow: add the operation to spec if it is not there, implement
it, run conformance.
**Done when** conformance runs and fails the build on all targets `ci-audit`
marked runnable, and the workflow is written down where someone adding an
architecture will find it.
## Track D — Spec content
Writing spec functions, docs, and bodies. The shared bottleneck the other tracks
draw on.
### `doc-explore` — Doc-mechanism exploration on a hard family
**Done:** [x] · **Needs:** — · **Blocks:** `spec-common` · **Risk:** `risk-doc-template-ceiling`
As of CL 831485, roughly one doc line in eight in `categories.yaml` carries
shape-specific literals — `[x0, x1, x2, x3, ...]`, `128-bit`, `0b_11_01_00_10` —
concentrated in `AddSub`'s `Concat*Pairs` and `Moves`'
`concatSelectedConstant*`. Those docs cannot be produced by substituting `{vL}`
into one template; each shape needs its own text.
The expected mechanism is multiple spec functions sharing one `//specgen:name`,
which specgen supports because `funcKey` is `(recv, name)` and the constraint
sets are disjoint. **Verify that before bulk migration**, not after: discovering
it does not scale once 150 easy docs are written means retrofitting all of them.
**Commits.** (1) Migrate `ConcatAddPairs` / `ConcatSubPairs` into spec, docs
included; their text differs between 16/32-bit and 64-bit elements. Verify via
`cmd/specls`. (2) Migrate the `Grouped` variants, whose docs vary by element size
*and* by the 128-bit grouping — two axes, and the real test of the mechanism. (3)
`ShiftSaturated`, `ShiftAllRight`, `ShiftRight`, whose docs differ between signed
and unsigned elements; `categories.yaml` selects the variant with a `signed:`
join key, which this retires. (4) If the mechanism proves insufficient, extend it
(conditional doc text is the likely shape) and document the extension in
`internal/spec/doc.go`.
`concatSelectedConstant*` is harder still — four spec functions for one API name
— but it is *unexported*, so it sits outside `specfill`'s scope per §1.7 and is
undecided until `compiler-ops`. Reach for it only if `compiler-ops` has landed
and you want a sterner test.
**Done when** `cmd/specls` reproduces the existing doc text for all three
families, or the doc mechanism has been extended so that it can.
### `spec-common` — Spec the common operations
**Done:** [x] · **Needs:** `width`, `doc-explore` · **Blocks:** `validate-amd64`, `comments-yaml`, `spec-all` · **Prefers:** `specfill` · **Risk:** `risk-ref-impl-effort`
The method set `midway/comments.yaml` documents: its `.common_methods` block
less `String` and `Len` — `Add`, `Sub`, `Min`, `Max`, `And`, `Or`, `MulAdd`, the
compares, and the rest of the portable core. `SPEC-STATS.txt` tracks it as
"portable core", which is this task's completion bar. Highest leverage in the
plan — these are the easiest reference implementations to write correctly, they
unlock `comments-yaml` (the worst doc divergence, in the widest-audience
package), and they are exactly what a new architecture implements first.
The file's keys are not all operations: besides the method docs, it documents
vector *types*, functions and templates, and carries receiver names keyed to the
shared block. Only the methods/functions are in scope here, and this excludes
non-spec residue (`String`, etc, §1.7).
**Commits.** Start by sampling ~10 operations at random and timing how long it
takes to implement them, to calibrate `risk-ref-impl-effort` before committing
to the rest. Then one commit per family (arithmetic, bitwise, min/max, compares,
multiply-accumulate), each adding the spec functions with docs and bodies and
confirming `cmd/specls` output.
**Done when** every *method* name in `midway/comments.yaml` other than non-spec
residue resolves to a spec function, with a real body, unblocking
`comments-yaml`.
### `spec-all` — Bulk spec migration
**Done:** [ ] · **Needs:** `spec-common`, `convert-lo`, `mask-bits` · **Prefers:** `specfill`, `fill-enforce` · **Blocks:** `template-docs` · **Risk:** `risk-ref-impl-effort`
The remaining exported operations, prioritized by the names produced by more
than one generator (SPEC-STATS.txt §1) — highest drift risk, and each migration
fixes several files at once. Best-effort bodies are acceptable; `panic("not
implemented")` is safe because conformance skips it. A *wrong* body is worse
than no body.
Two denominators, both useful. Against the API: the `(receiver, name)` pairs
SPEC-STATS.txt §1 shows uncovered, which is what the scoreboard tracks. Against
the input being retired: `categories.yaml`'s exported operation names, less the
ones `spec-common` handles. They are different counts of different things — one
`categories.yaml` name can expand to many API names — so quote whichever the
audience needs and say which.
**Why `convert-lo` and `mask-bits` gate this.** Only the Converts and LoadStore
families actually depend on them — `convert-lo` may rename `ExtendLo*` /
`ConvertLo*`, and `mask-bits` may rename the scalable mask bitmap pair, and
renaming after the docs are migrated means doing them twice. If either is still
open when this task starts, the other families are unaffected and are the place
to begin.
**Commits.** One per operation family, mirroring the `ops/*/` grouping. Each
spec function gains `//specgen:commutative` where the operation is commutative;
the directive already exists and is parsed by `specgen`. Drop any
`comments.yaml` entries spec gains as they are covered. Include all regenerated
outputs in each commit. Once `gen-docs` has landed, each family's regeneration
rewrites that family's docs, so review the diff to ensure all doc comment
content is equivalent: any dropped text is either something spec should say or
an implementation note missing its prefix.
**Done when** `specfill` reports no remaining API declarations uncovered by spec,
other than the permanent residue named in §1.7 (`String`, `Len`).
## Track E — Retiring the old inputs
Retiring the other documentation sources, then `categories.yaml` itself.
### `doc-triage` — Hand-written doc conflict triage
**Done:** [ ] · **Needs:** `specfill` · **Prefers:** `spec-common` · **Blocks:** `handwritten-fill` · **Risk:** `risk-silent-doc-loss`
Make it safe for `handwritten-fill` to rewrite hand-written files, by emptying
the conflict set first.
A conflict is a declaration where spec has a doc *and* the hand-written file has
different spec-owned paragraphs (§1.6). Those are the only sites `-w` would
overwrite: where spec has no entry, the declaration is left alone. Every
conflict therefore resolves one of two ways — the prose says something spec
should say (move it into spec), or it is about how a target implements the
operation (make it an implementation note by giving it one of §1.6's prefixes,
and `specfill` never touches it). Neither outcome loses text, which is what makes
`-w` safe afterwards.
Run `specfill -diff` over the hand-written files — every doc comment it would
rewrite is a conflict — and sort each conflict into one of those two.
SPEC-STATS.txt §3 breaks the conflict count down by file, with
`simd_emulated.go` dominating and a long tail of `archsimd/slicepart_*.go`,
`shuffles_amd64.go`, and `extra_amd64.go`. Some implementation text there has no
prefix today — `extra_amd64.go`'s "This method compiles to VPTEST x, x" is a
`Performance:` note — and the same pass fixes check (a) violations and note
spellings in these files.
The `slicepart_*.go` files matter most: they declare the `Load*Part` family spec
*already* covers via `LoadZPart`, so their prose conflicts on day one — and one
of them documents `LoadUint32x4Part` as loading "a Int32x4".
**Done when** `specfill -no-fill-names -diff` is empty and `specfill` reports
zero check (a) violations.
### `comments-yaml` — Drop spec-covered docs from `midway/comments.yaml`
**Done:** [x] · **Needs:** `gen-docs`, `spec-common` · **Prefers:** `fill-enforce`
Every method or function-template entry spec covers is dropped from
`comments.yaml`. At the time of this task (after `spec-common`) that is the
portable core less non-spec residue (`String`, etc, §1.7), plus the
Load/LoadPart/Broadcast templates. Entries spec gains afterwards (in `spec-all`)
are dropped when spec gains them.
**Commits.** (1) Confirm spec covers the portable core; fill gaps. (2) Delete
the covered entries from `comments.yaml`. (3) Run `specfill -w` over
`simd/simd_emulated.go` — this overlaps `handwritten-fill`; sequence whichever
lands first.
**Done when** `comments.yaml` has no entry for any operation spec covers, midway
sources those docs through `specfill`, and `simd_stubs.go` and
`simd_emulated.go` carry identical spec-derived docs for them.
### `handwritten-fill` — Enable `-w` on hand-written files
**Done:** [ ] · **Needs:** `gen-docs`, `doc-triage` · **Risk:** `risk-silent-doc-loss`
Turn on doc rewriting for hand-written files, once `doc-triage` has emptied the
conflict set — at which point `-w` rewrites only docs that already match what it
would write.
Two guards, both on `cmd/specfill` only. This does **not** change `gentools`'
shared `-w`, which every generator uses:
- `specfill -w` refuses to run when the report has any findings, or if there are
uncommitted changes, so its diff is the complete story of what it did.
- `specfill -w` never runs in CI; CI runs report mode (`fill-enforce`).
**Done when** `specfill -w` over the hand-written files is a no-op on a clean
tree.
### `template-docs` — Retire generator-supplied API docs
**Done:** [ ] · **Needs:** `gen-docs`, `spec-all` · **Blocks:** `cat-rehome`
Delete the doc strings embedded in wasmgen's and tmplgen's templates, and the
`documentation:` fields on exported `categories.yaml` entries, leaving
`specfill` to supply them. Mechanical once `spec-all` has covered the operations
those generators emit — but not automatic, because their templates interleave
prose with the implementation notes that stay with the generator (§1.6).
**Commits.** One per generator, stripping the prose and keeping the notes; one
for the `documentation:` fields on exported `categories.yaml` entries. Each
includes its regenerated output. Then stop setting `AllowDocRewrite` in the
generators, so a generator that emits API prose again fails generation.
**Done when** no generator's source contains API prose, no exported
`categories.yaml` entry has `documentation:`, regeneration is byte-identical
to the previous tree, and the generators run without `AllowDocRewrite`.
### `cat-rehome` — Re-home `categories.yaml` non-doc fields
**Done:** [ ] · **Needs:** `template-docs` · **Blocks:** `compiler-ops-impl`, `sig-from-spec` · **Risk:** `risk-cat-unknowns`
Empty the exported entries of `categories.yaml` of everything but `go:` and
`commutative:`, by sending each remaining field where §1.3's inventory says it
belongs. `template-docs` has already removed `documentation:`; this is the rest.
Compiler-only entries are untouched, for `compiler-ops-impl`.
Expect surprises. §1.3's audit was static, and the unifier's behavior on field
combinations was not exhaustively traced — `regexpTag`'s role is still
unverified (§1.3). Mitigating factor: `go run . -diff` makes every step
verifiable byte-for-byte against checked-in generated output.
**Commits.** Follow §1.3's disposition column, one field per commit, confirming
generated output is unchanged at each step: (1) delete `regexpTag` and `signed`
from `categories.yaml` and from `go_*.yaml`, expecting most `go_*.yaml` uses to
be inert already; (2) move `constImm` into `go_amd64.yaml`; (3) convert the
`out`/`elemBits`/`bits` entries into spec function splits, which `spec-all`
should already have done — verify.
**Done when** every exported entry in `categories.yaml` has only `go:` and
`commutative:` keys; compiler-only entries are untouched, for
`compiler-ops-impl`. Generated output is byte-identical throughout.
### `compiler-ops-impl` — Implement compiler-only ops representation
**Done:** [ ] · **Needs:** `compiler-ops`, `cat-rehome` · **Blocks:** `sig-from-spec`
Implement whatever `compiler-ops` decided, for the compiler-only entries —
those that produce no exported declaration — and the `noTypes` / `noGenericOps`
switches. No design work should remain at this point; if it does, `compiler-ops`
was not finished.
These operations have no API surface — they exist only as SSA ops and lowering
rules — so nothing here touches `specfill`, spec, or the public packages. It is
entirely about giving simdgen somewhere other than `categories.yaml` to read them
from.
**Done when** no compiler-only entries remain in `categories.yaml`
(SPEC-STATS.txt §4) and generated output is byte-identical.
### `cat-delete` — Delete `categories.yaml`
**Done:** [ ] · **Needs:** `sig-from-spec`
Delete `ops/*/categories.yaml`, the `!import` stub that loads it, and its
handling in `simdgen/main.go`. By this point every field it carried has been
re-homed (`cat-rehome`), its compiler-only entries have moved
(`compiler-ops-impl`), and its signatures come from spec (`sig-from-spec`), so
this should be pure deletion.
**Done when** `categories.yaml` no longer exists, simdgen runs without it, and
generated output is byte-identical.
## Track F — simdgen internals
The signature refactor and, eventually, sourcing signatures from spec.
### `sig-refactor` — simdgen signature refactor
**Done:** [ ] · **Needs:** — · **Blocks:** `sig-from-spec` · **Risk:** `risk-sig-refactor-convergence`
Make the Go signature a first-class value on `Operation` instead of an emergent
property of `shape()` → `classifyOp()` → a template name. Ideally shaped so it is
directly comparable to `specgen.Func`'s `Recv`/`In`/`Out`.
This collapses the stub templates (§1.1 counts them) to roughly three, because
operand permutation and parameter naming become data rather than template
identity; the `_21`, `_231Type1`, `_31` suffixes become an explicit permutation.
It is also the empirical test of whether the unification approach can be kept.
The refactor assumes the signature is expressible as data without a new pile of
special cases; the quirk fields and `simdgen only supports…` panics catalogued
in §1.1 are mostly *upstream* of the signature (XED inconsistency) or
*downstream* (register shapes, rules), so the seam looks thin — but that is a
prior, not a measurement. If it does not converge, that is the evidence for
revisiting §1.1, earned cheaply on the few hundred lines §1.1's layer table
isolates.
**Commits.** (1) Introduce the signature type and a `GoSignature()` method
covering the simplest classes (`op1`, `op2`, `op3`). (2) Extend to immediate
variants (`op*Imm*`), which may be fiddly. (3) Extend to permuted and
`VecAsScalar` variants. (4) Rewrite the stub templates to consume the signature
value; delete the redundant templates.
**Done when** `go run . -diff` shows byte-identical output at every commit.
### `sig-from-spec` — simdgen sources signatures from spec
**Done:** [ ] · **Needs:** `sig-refactor`, `cat-rehome`, `compiler-ops-impl` · **Blocks:** `cat-delete`
simdgen looks up the spec function by `(recv, name)` and asserts signature
equality, reporting a structured diff on mismatch. **Key the lookup on
`(recv, name)` only** — spec's own uniqueness key — and assert the signature
separately. Keying on the signature makes a divergence indistinguishable from
"not migrated yet", which is the failure this whole effort exists to catch.
This is where `sig-refactor` pays off: the lookup compares against the signature
value that refactor introduced, rather than against a rendered template.
**Commits.** (1) Look up the spec function and assert equality, failing the
generator run on mismatch. (2) Delete whatever signature information in
`go_GOARCH.yaml` is now redundant. (3) Switch simdgen to reading commutativity
from spec's `//specgen:commutative` directive and delete `commutative:` from
exported `categories.yaml` entries.
**Done when** simdgen derives every exported signature it emits from spec,
commutativity comes from spec, and generated output is byte-identical.
---
# Part 4 — Risks and unknowns
Every risk is owned by a task, and its substance is stated there. This is the
index. `risk-concrete-width`, `risk-impl-defined-behavior` and
`risk-ref-impl-effort` are the expensive ones: each reaches every spec body, so
each is far cheaper to settle before the bodies are written than after.
| | Risk | What it is | Owner |
|---|---|---|---|
| **[ ]** | `risk-spec-defects` | Known spec defects | `convert-lo`, `mask-bits` |
| **[x]** | `risk-concrete-width` | Executing spec at a concrete width | `width` |
| **[ ]** | `risk-impl-defined-behavior` | Implementation-defined behavior | `impl-defined` |
| **[x]** | `risk-doc-template-ceiling` | Doc-template ceiling — the fallback is a mechanism change affecting every doc written so far | `doc-explore` |
| **[ ]** | `risk-ref-impl-effort` | Reference-implementation effort — every migrated operation needs a body that is *correct*, because conformance tests against it; the distribution of difficulty is unknown | `spec-common`, `spec-all` |
| **[x]** | `risk-conformance-env` | Conformance execution environment — can CI run arm64/SVE/wasm, and at what cost? | `ci-audit` |
| **[ ]** | `risk-sig-refactor-convergence` | `sig-refactor` may not converge — if it does not, §1.1 needs revisiting | `sig-refactor` |
| **[ ]** | `risk-silent-doc-loss` | Silent doc loss — under §1.6, text without a note prefix is spec-owned, so implementation text nobody prefixed is dropped on the first rewrite; the guards are review of every rewrite's `-diff` and, once generators emit no prose of their own, `AllowDocRewrite` | `gen-docs`, `doc-triage`, `handwritten-fill` |
| **[ ]** | `risk-cat-unknowns` | Unknown unknowns in `categories.yaml` — the field audit was static | `cat-rehome` |
---
# Appendix A — Regenerating the two generated files
## `SPEC-STATS.txt`
`cmd/specstats` generates [SPEC-STATS.txt](SPEC-STATS.txt), the transition
scoreboard:
```sh
cd $GOROOT/src/simd/archsimd/_gen
go run . -w # (if necessary) ensure generator outputs are up-to-date
go run ./cmd/specstats # print the report
go run ./cmd/specstats -diff # see what's changed
go run ./cmd/specstats -w # rewrite SPEC-STATS.txt in place
```
It loads `simd` and `archsimd` through `go/packages` for each build
configuration, taking syntax only — no type-checking, per §1.4 — and reads
`simd/internal/spec` through `specgen.Load`. It needs a recent Go toolchain and
up-to-date output from the SIMD generators, but does not itself require any ISA
data. Commit the regenerated file alongside whatever change moved the numbers;
the diff is the record of what moved.
It reports only figures that measure how far along the transition is: every one
is a fraction heading for 100% or a count heading for zero, tagged with the
tasks that move it; indented, untagged numbers are context breaking down the
figure above them. If you add a figure, give it a denominator and a direction,
or leave it out.
One thing it deliberately does not do: it classifies divergence (§5) but fails
on nothing; enforcement is `specfill`'s job. `specfill` should absorb the
comparison half of this program rather than leave two implementations to drift.
For one-off questions the report does not answer, `cmd/specls` prints the API
spec currently expands to:
```sh
go run ./cmd/specls # the whole expansion
go run ./cmd/specls -f Add # one spec function
go run ./cmd/specls -f Add -trace # why it expands the way it does
```
## `SPEC-TASKS.dot`
The task graph is drawn from Part 2's two tables by `cmd/spectasks`:
```sh
go run ./cmd/spectasks # print the graph
go run ./cmd/spectasks -w # rewrite SPEC-TASKS.dot in place
dot -Tsvg SPEC-TASKS.dot -o /tmp/tasks.svg # render to SVG
go run ./cmd/spectasks | grep -v ^-- | xdot - # view the current graph
```
It reads nothing but this file. Two things it does beyond drawing:
It **checks**, from `go test ./cmd/spectasks`. Dependencies are stated three
times here — Part 2's table, Part 3's `Needs:`/`Prefers:` lines, and Part 4's
`Blocks:` lines, which are the same edges reversed — and the test fails if the
three disagree, if the table is not topologically sorted, if a task is in no
track or in two, or if an edge points at a task that does not exist. A second
test fails if SPEC-TASKS.dot no longer matches the document, so an edit that
skips the regeneration step is caught too. The command itself refuses to write a
graph from an inconsistent document, but the test is where the problems are
reported.
It **derives the critical path** rather than restating one: a task is on it when
its longest chain in plus its longest chain out spans the whole graph. That
keeps every longest chain, not one picked by hand, so parallel roots of equal
depth are all marked.