| # Transitioning the Go SIMD generators to `internal/spec` | |
| This plan lays out how to transition the myriad generators and hand-written | |
| sources of the SIMD API from disjoint, manually-enforced consistency, to a | |
| single underlying executable API specification that provides all method | |
| signatures, documentation, and checkable semantics. | |
| See [README.md](README.md) for background on the SIMD packages, the "one name, | |
| one meaning" rule, the generators, how simdgen works, what spec and specgen are, | |
| and how to run things. | |
| ## Table of contents | |
| - Part 1: Decisions and rationale | |
| - Part 2: Task graph | |
| - Part 3: Task details | |
| - Part 4: Risks and unknowns | |
| - Appendix A: Regenerating the two generated files | |
| ## The plan in brief | |
| `internal/spec` — generic Go functions describing each operation — becomes the | |
| single hand-written description of the SIMD API. Today the same operation is | |
| described in six places, and they have already drifted from each other (§1.5). | |
| Three things have to happen, and they are largely independent: | |
| 1. **Build the `specfill` checker.** One AST pass that injects spec's docs into | |
| generated and hand-written source and verifies signatures against spec. Types | |
| can be enforced almost at once — SPEC-STATS.txt §2 shows zero type-level | |
| mismatches between spec and the shipped API. | |
| 2. **Write the spec functions** (`spec-common` and `spec-all`). Coverage is in | |
| SPEC-STATS.txt §1. This is the bottleneck the other | |
| tracks draw on. | |
| 3. **Retire the old inputs**: the other doc sources, then `categories.yaml`, at | |
| which point the unifier no longer defines the API — it only expands one arch | |
| entry into the machine instructions matching it. | |
| Behavioral conformance against `internal/simdref` (track C) runs in parallel. | |
| Beyond its own `ci-audit`, it depends only on spec coverage and on the `width` | |
| and `impl-defined` decisions. | |
| ## Monitoring plan progress | |
| Progress on this transition is tracked in two ways: | |
| **Task completion** is tracked directly in this document via the `[ ]` | |
| checkboxes on each task and risk. The task table in part 2 summarizes all task | |
| completion in one place. | |
| [SPEC-TASKS.dot](SPEC-TASKS.dot) shows a visual representation of the task graph | |
| and completed tasks. It is generated directly from this document by running `go | |
| run ./cmd/spectasks -w` from `_gen`. This generator also checks the consistency | |
| of this doc, and the test in `./cmd/spectasks` enforces doc consistency in CI. | |
| Rerun the generator after modifying any task metadata. | |
| **Measurements** live in [SPEC-STATS.txt](SPEC-STATS.txt), which tracks spec | |
| coverage of the API, signature and name agreement, documentation sources, and | |
| retirement of the old inputs. It is generated by running `go run ./cmd/specstats | |
| -w` from `_gen`. Every figure in that file is a fraction heading for 100% or a | |
| count heading for zero, tagged with the tasks that move it. Indented, untagged | |
| numbers are context breaking down the figure above them. Part 1 of this doc | |
| quotes measurements that motivated specific decisions, stamped with the CL at | |
| which they were measured; those are frozen and do not need refreshing. | |
| ## How to implement a task | |
| Part 2 tells you which task to pick up: the next one whose prerequisites are all | |
| ticked. [SPEC-TASKS.dot](SPEC-TASKS.dot) draws the same thing if a picture | |
| helps. The commit sequences in Part 3 are guesses made before the work started; | |
| deviate when the code tells you to, but say so in the CL description. | |
| Tasks and risks carry `[ ]` checkboxes. When you resolve one, tick the box, | |
| update the task text to reflect the reality of the implementation so it remains | |
| useful to reference when performing follow-on tasks (you can drop or summarize | |
| the exact commit sequence), and regenerate [SPEC-TASKS.dot](SPEC-TASKS.dot). | |
| Commit this as its own CL on top of the CLs that implement the task, and include | |
| justifications and reasoning for any changes to the task text in the commit | |
| message. | |
| ## Key concepts and terms | |
| **`specfill`** is the tool this plan builds first and then leans on. It is one | |
| AST pass over Go source that injects spec-derived doc comments and parameter | |
| and result names, and verifies signatures against spec. It exists as both a | |
| package that can be integrated into generators and a standalone tool: | |
| - **`specdoc.Fill`** is the library entry point — Go source in, Go source out: | |
| `func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`. | |
| If verification checks fail, it returns a `*Report` as an error. Generators | |
| call it on their `gentools` buffer before writing. | |
| - **`cmd/specfill`** is the `gentools`-based command wrapping it, for running | |
| over hand-written files. Where this document writes `specfill -w`, that is | |
| the command; where it writes `specfill`, that is the library function. CI | |
| enforcement is a `go test` in this package. | |
| These package and command names are suggestions; the split between library and | |
| command is not. | |
| **Target** means one of amd64, arm64 (NEON), arm64 SVE, or wasm. See README.md. | |
| **Spec operation** means one function in package `spec`. Each expands across | |
| element types, widths, and templated names into many API names and many | |
| `(receiver, name)` pairs, so it is the grain spec is *authored* at — and the | |
| only grain at which "implemented nowhere" means anything (SPEC-STATS.txt §5). | |
| **Operation** (unqualified) means the API-level operation a name denotes, | |
| whichever of spec, `categories.yaml`, or generated source describes it. A | |
| `categories.yaml` entry is called an *entry*; one operation name can have several. | |
| **Granularity.** Spec coverage is measured at three granularities: | |
| `(target, receiver, name)` — every exported declaration, counted once per build | |
| configuration; `(receiver, name)` — collapsed across targets, the primary | |
| denominator since spec describes an operation once; and `name` — collapsed across | |
| receivers too, coarser and misleading when a name exists on different receiver | |
| sets in spec and the API (SPEC-STATS.txt §5). SPEC-STATS.txt §1 reports all | |
| three. `String` and `Len` are excluded throughout (§1.7). | |
| --- | |
| # Part 1 — Decisions and rationale | |
| These are the load-bearing decisions. Each was contested during design. The | |
| reasoning is recorded so it need not be re-litigated, while also making it | |
| easier to revisit decisions if new evidence arises. | |
| Measurements in this section were taken at CL 831485 unless otherwise noted. | |
| They should NOT be updated unless decisions need revisiting. | |
| ## 1.1 We are not rewriting simdgen | |
| The temptation is real: simdgen has 28 stub templates whose *names* encode | |
| arity, operand permutation, and immediate placement (`op3_231Type1`, `op4_31`, | |
| `op2Imm8_II`); 42 fields across `RawOperation` (23) and `Operand` (19); and 27 | |
| bail-outs whose messages begin "simdgen only supports" or "simdgen does not". | |
| But the layer accounting says the mess is small and the value is large (at | |
| CL 831485; Go counts include `_test.go`; "signature derivation + emission" is | |
| `simdgen/gen_*.go`, `godefs.go`, `arch.go`, and `types/operation.go`): | |
| | Layer | Lines | Affected by "was unification wrong?" | | |
| |---|---|---| | |
| | Platform loaders | 5,100 | No — irreplaceable | | |
| | Unifier engine | 3,169 | Partly | | |
| | Signature derivation + emission | 4,715 | The ~500 messiest lines, yes | | |
| | `categories.yaml` | 1,284 | Deleted by this plan anyway | | |
| | `go_GOARCH.yaml` | 4,475 | No | | |
| The part that would be thrown away if unification were the wrong call is the ~500 | |
| messiest lines of signature derivation, against irreplaceable platform loaders | |
| several times that size. | |
| A few thousand lines of hand-written YAML drive tens of thousands of lines of | |
| generated API and compiler backend across four targets. That ratio is good. | |
| What is *not* earning its keep is unification's role as **API definer** — the | |
| `categories.yaml` ↔ `go_GOARCH.yaml` join — and this transition removes that role | |
| by construction. The unifier's remaining job, expanding one arch entry into every | |
| matching concrete instruction variant against the ISA database, it does well. | |
| ## 1.2 Spec is not a unifier input | |
| Feeding spec into unification via `DefBuilder` is easy and wrong. Spec's | |
| structure would be flattened into the unstructured value language the moment it | |
| crossed the boundary, recreating the exact failure mode we are escaping. Instead, | |
| simdgen consumes `[]*specgen.Func` as a typed Go table. | |
| Furthermore, spec is inherently only aware of the structure of the Go SIMD API, | |
| but the inputs to the unifier are in platform terms. Much of the output of the | |
| unifier is information on how to map the platform structure into the Go SIMD | |
| API, so we can't bring spec in until *after* that mapping has been processed. | |
| ## 1.3 What belongs in spec | |
| The test is: **is it a checkable claim about the spec's own semantics?** | |
| - `commutative` — yes. `Add(x,y) == Add(y,x)` is testable against the executable | |
| body, and spec is the only place that *can* check it. | |
| - `noGenericOps`, `noTypes` — no. Claims about generator output. | |
| - `constImm` — no. A claim about x86 encoding. | |
| - `regexpTag`, `signed` — no. Claims about the unifier. | |
| The test is self-policing: anything that passes it falls out of the conformance | |
| testing we want anyway. Note it admits facts the *compiler* consumes — the wrong | |
| line to draw is "API vs compiler"; the right one is "intrinsic to the operation | |
| vs incidental to the mechanism or platform." | |
| For `commutative`, no name in `categories.yaml` currently needs | |
| platform-specific commutativity, so a function-level directive in spec is | |
| sufficient today. However, there may be corner cases we haven't uncovered; | |
| notably, we have evidence that `Min`/`Max` on floats may not be commutative on | |
| all platforms. It may suffice to simply not mark these operations as | |
| commutative, and to document when an operation is not commutative for surprising | |
| reasons. | |
| Applying the test to `categories.yaml`'s fields (at CL 831485; 217 entries, 195 | |
| unique operation names, 163 exported): | |
| | Field | Count | Transition plan | | |
| |---|---|---| | |
| | `go` | 217 | → spec function name | | |
| | `commutative` | 217 | → `//specgen:commutative` directive; `categories.yaml` copy removed by `sig-from-spec` | | |
| | `documentation` | 211 | → spec doc comment; removed by `template-docs` | | |
| | `regexpTag` | 25 | unification join key; role unverified (see `risk-cat-unknowns`) | | |
| | `constImm` | 22 | → sideways into `go_amd64.yaml` (x86 encodings) | | |
| | `out` | 14 | join key used only for selecting a doc variant; → multiple spec funcs sharing one `//specgen:name` | | |
| | `elemBits` | 14 | same | | |
| | `bits` | 4 | same | | |
| | `signed` | 12 | same | | |
| | `noGenericOps` 13 / `noTypes` 6 | 13 entries | **open question** — compiler-only ops | | |
| Task `cat-rehome` tackles most of the "transition plan" column. | |
| `compiler-ops-impl` handles the `noGenericOps` / `noTypes` entries. | |
| ## 1.4 `specfill` parses; it does not type-check | |
| `ops_amd64.go` and `types_amd64.go` have minimal imports, and no exported | |
| signature in either package uses a qualified type, so `go/parser` is sufficient — | |
| no `go/types`, no `packages.Load`, no GOEXPERIMENT wrangling, no cross-module | |
| loading of GOROOT std packages. | |
| This is better than a workaround. Type-checking sees one build configuration at a | |
| time and would make the cross-architecture comparison *harder*. Parsing all files | |
| regardless of build tag yields `(Int32x4) Add` from `ops_amd64.go`, | |
| `ops_arm64.go`, and `ops_wasm.go` in a single pass, which is precisely the | |
| comparison we want. `apisum` already does this, so it is proven on this API. | |
| `cmd/specstats` (Appendix A) takes a different route to the same end: it loads | |
| syntax through `go/packages` once per build configuration, because it reports | |
| per-target figures. It still does not type-check, and `specfill` does not need | |
| per-configuration loading. | |
| ## 1.5 Two-pass doc injection, and why the "disadvantage" is the point | |
| The motivation is doc divergence. At CL 831485 there were **six independent | |
| documentation sources** — `categories.yaml`, wasmgen inline strings, tmplgen | |
| inline strings, `midway/comments.yaml`, hand-written `simd_emulated.go`, and | |
| hand-written `archsimd` files — and none agreed with any other. All 90 | |
| `(receiver, name)` pairs documented in both `simd` and `archsimd` had different | |
| doc text, and within the portable package, `simd_stubs.go` (midway-generated) and | |
| `simd_emulated.go` (hand-written) had drifted on 40 of 336 mirrored methods. | |
| `Add` shipped with three independent wordings: | |
| | Where | Text | Source | | |
| |---|---|---| | |
| | `ops_amd64.go`, `ops_sve.go` | *Add adds corresponding elements of two vectors.* | `categories.yaml` | | |
| | `ops_wasm.go` | *Add returns the result of adding x and y, elementwise.* | wasmgen, inline | | |
| | `simd_stubs.go`, `simd_emulated.go` | *Add returns the element-wise sum of x and y.* | `midway/comments.yaml` | | |
| 124 method names are produced by more than one generator — the easiest place for | |
| disagreement to creep in. simdgen accounts for the majority of declarations, but | |
| tmplgen, wasmgen, midway, and hand-written code own the rest, and those sets | |
| overlap — any instrument that only sees simdgen misses a substantial fraction of | |
| the problem (SPEC-STATS.txt §1 has the per-generator counts). | |
| Generators pass their generated code to `specfill`, which checks signatures and | |
| injects docs before writing to disk; in the end state, generators emit | |
| declarations and implementation notes (§1.6) but no API prose of their own. The | |
| alternative — having specgen emit the whole declaration, as `refgen` does — was | |
| rejected because it would make agreement with spec true by construction and | |
| leave nothing to verify. A generator whose internal model of the API had drifted | |
| would be silently papered over. Because the template still produces the | |
| signature, `specfill` sees two independent witnesses to the same declaration and | |
| can compare them: docs in, signatures verified, coverage reported, one AST walk. | |
| Consequences: | |
| - **`specfill` is an override, not a replacement.** Generators keep emitting | |
| their current docs; `specfill` replaces the spec-owned paragraphs only when spec | |
| knows the operation. Fallback is therefore automatic and no flag day is | |
| required. Prose-less emission is the *end state* reached when a generator's doc | |
| source empties, not a precondition. | |
| - **wasmgen and tmplgen need only a call before flush**, plus making the notes | |
| they emit follow §1.6's prefixes. | |
| - **Do not hook this into `gentools.Files.Flush()`.** `gentools` is a general | |
| utility; coupling it to specgen would make every consumer depend on | |
| `golang.org/x/tools/go/packages` and a loadable spec tree. A narrow hook | |
| mechanism in gentools plus a few lines per generator keeps the boundary. | |
| ## 1.6 The doc-ownership convention | |
| `specfill` rewrites part of an existing doc comment, so it needs a rule for which | |
| part it may touch, and running it twice must be idempotent. The obvious rule, | |
| "everything above the first bare `//`", doesn't work. At CL 831485 only 2,759 of | |
| 3,920 documented declarations have that separator. And spec's own docs contain | |
| blank comment lines: `Add`'s is a summary, a `//`, then `z[i] = x[i] + y[i]`. | |
| Hence, using a bare `//` would not be idempotent. | |
| **The rule.** A doc comment is spec-owned paragraphs, followed by | |
| "implementation notes", followed by directive comments (`//go:fix inline` and | |
| similar). Spec-owned paragraphs describe the behavior of an operation. | |
| Implementation notes are details of how one target implements an operation. The | |
| sections must appear strictly in this order, and the tools will validate this. | |
| An *implementation note* is a paragraph that starts with one of a fixed set of | |
| prefixes: `Asm:`, `CPU Feature:`, `Emulated[,\n]`, `Deprecated:`, | |
| `Performance:`, or `Note:`. The match is exact and case-sensitive. Only the | |
| first line of a paragraph is checked, so a note can combine prefixes (`Asm: | |
| VPADDD, CPU Feature: AVX`) and wrap freely. | |
| **Checks.** Strict ordering is necessary for idempotence, and the tools perform | |
| the following checks to ensure we maintain it: (a) In an existing comment, | |
| spec-owned paragraphs (those without an implementation note prefix) must appear | |
| strictly before implementation notes (those with an implementation note prefix), | |
| which must appear strictly before directive comments. (b) Spec's docs may not | |
| contain a line starting with a note prefix, and spec stores them in canonical | |
| `go/doc/comment` form. | |
| **Rewrite.** `specfill` rewrites a comment only when spec has an entry for the | |
| declaration and check (a) passes. The new comment is the spec doc, then a blank | |
| line if there are implementation notes, then those notes unchanged, then the | |
| directives unchanged. In every other case the comment is left alone. This is | |
| idempotent: by (b) the injected spec doc never looks like a note, so the next run | |
| splits the comment at the same place and produces the same bytes. Idempotence | |
| can't catch text lost on the *first* run, though, so as we transition to spec, | |
| generated docs for newly covered operations must be reviewed carefully. | |
| **Why prefixes, not markers.** The alternative was bracketing `specfill`'s region | |
| with directive comments (`//spec:doc` … `//spec:end`). Those are invisible in | |
| godoc and safe by default — no marker, nothing touched — but they add lines to | |
| every declaration and say nothing about *why* text is outside spec. Prefixes | |
| reuse the shape the notes already have, work the same in generated and | |
| hand-written files, and force each piece of non-spec text to say what kind of | |
| note it is. The cost is that unmarked text defaults to spec-owned, which is why | |
| every rewrite is reviewed as a diff, and why generators, once they emit no API | |
| prose of their own, may not emit spec-owned text at all (see `specfill`). | |
| ## 1.7 Errors versus tolerated gaps | |
| At CL 831485, of the 782 `(receiver, name)` pairs declared for more than one | |
| target, not one had divergent parameter or result *types* — but 72 disagreed on | |
| parameter names and 16 on result names. So spec is *preventive, not remedial*: | |
| there is no backlog of type-level fixes, and incremental migration is not racing | |
| decay. | |
| - **Signature mismatch is an error.** For an operation spec knows about, a | |
| disagreement is a bug in spec or in a generator. | |
| - **A missing spec function is tolerated** until the transition completes. | |
| - **Parameter and result names are checked** where the declaration names them | |
| (unnamed ones are filled from spec; §1.8), for two reasons: they appear in | |
| documentation and so must be consistent with the documentation's source, and | |
| several past bugs have come from confused or swapped argument names. | |
| - **`specfill` considers exported declarations only.** This is what makes the | |
| tolerated-miss list converge: all three `ops_internal_*.go` files (amd64, | |
| arm64, sve) contain zero exported declarations, so every compiler-only | |
| operation falls outside `specfill`'s view without needing a category. The | |
| permanent residue is small and enumerable: the `String`, `Len`, `ToArch` | |
| methods and `xFromArch` functions. | |
| ## 1.8 Named results | |
| Spec names its results and its docs reference those names (`z[i] = x[i] + | |
| y[i]`). Almost nothing in the shipped API names its results (SPEC-STATS.txt §2). | |
| When spec's doc becomes archsimd's doc, the references to `z` in the docs won't | |
| make sense unless we carry over spec's result names as well. | |
| Decision: **where spec covers a declaration, its parameter and result names | |
| come from spec**, so they agree with the doc that also comes from spec. Naming | |
| every result is *not* a goal: a name is worth having where a doc or a body uses | |
| it, and naming the rest blindly serves no one. Declarations spec does not cover | |
| keep whatever names they have until `spec-all` covers them. | |
| `specfill` enforces this without touching bodies. It rewrites unnamed | |
| arguments/results in the API (including `_`) to match spec, and reports a | |
| mismatch error if both sides name an argument/result and the names disagree. For | |
| a bodyless declaration, all that matters is that the names agree with the doc. A | |
| body can refer to named arguments but not unnamed ones, so filling unnamed ones | |
| is safe, while named ones must be checked. SPEC-STATS.txt §2 counts the two | |
| cases separately, the same way. | |
| ## 1.9 Behavioral conformance is a parallel track | |
| Behavioral conformance has **no hard dependency on the API documentation | |
| track**. Both consume the same input — spec coverage — independently, and they | |
| catch disjoint bug classes: a swapped argument in a new architecture's `Sub` has | |
| an identical signature and identical docs, and only conformance sees it. | |
| The order for bootstrapping conformance matters. **Validate spec against amd64 | |
| first** — it is the most mature target, and every failure is a spec bug unless | |
| proven otherwise. There may be bugs or hardware-specific corner cases in the | |
| amd64 implementation, but bugs in spec are far more likely. Once spec and amd64 | |
| agree, **then flip**: spec becomes the oracle and gates arm64, wasm, SVE, and | |
| any architecture added later. Getting this backwards means conformance failures | |
| against correct implementations, and the obvious move is then to "fix" the | |
| implementation. | |
| Harness requirements implied by the bug classes we actually see: per-type | |
| edge-case inputs (0, ±1, MinInt, MaxInt, ±0, ±Inf, NaN, saturation boundaries) | |
| crossed with random vectors — uniform random will not reliably hit saturation or | |
| NaN corners; and explicit mask patterns, since an inverted mask sense passes | |
| random testing about half the time. | |
| **This is a path we have already gone down successfully.** | |
| `simd/internal/test_helpers` carries per-type input vectors — `float32s` through | |
| `uint8s`, plus `bools` for masks — built by cycling a hand-picked list of | |
| exactly those corners (±0, ±Inf, quiet and signalling NaN, `MaxFloat`, and the | |
| 8- and 16-bit saturation edges) up to a fixed length. In | |
| `archsimd/internal/simd_test/helpers_test.go`, the | |
| `forSlice`/`forSlicePair`/`forSliceTriple`/`forSlicePairMasked` helpers walk | |
| windows over them for the cross-product, `map1`/`map2`/`map3`/`mapCompare` build | |
| the expected result elementwise — the same shape a spec body has — and | |
| `grouped2` already handles the 128-bit grouping that gives `doc-explore` | |
| trouble. `checkSlicesLogInput` even takes a tolerance, which is a first cut at | |
| the vocabulary `impl-defined` has to settle. | |
| `conform-fixed` need not reuse that code literally, but it should not start over | |
| either: what it adds is generating the cases from spec rather than hand-writing | |
| them per operation. | |
| ## 1.10 Progress is measured by name, not by normalization | |
| `apisum` needs normalization because its job is to *test* whether names mean one | |
| thing. Once we enforce that they do, **the method name is the equivalence class**, | |
| exactly and for free: coverage is a set intersection, and `cmd/specstats` already | |
| computes it. No name normalization, and no complex pattern matching rules to | |
| maintain or throw away. | |
| Caveat, worth knowing rather than fixing: templated-name families like | |
| `LoadInt8x16`, `LoadInt8x32`, etc, that all come from one spec function inflate | |
| the denominator of any coverage metrics. That is arguably correct for measuring | |
| API *surface*. Reporting both numbers — "spec has N functions covering M names" | |
| — is more informative than either alone. | |
| --- | |
| # Part 2 — Task graph | |
| The 27 tasks fall into six **tracks**. Tracks are a way to divide the work among | |
| people and to make Part 3 navigable. They are not phases and don't imply any | |
| ordering on tasks. Tasks in each track generally cluster in the task dependency | |
| graph, but there are inter-track dependencies. | |
| | | Track | What it is | Tasks | | |
| |---|---|---|---| | |
| | **A** | Decisions | Human judgment calls; no prerequisites, all block downstream work | `width`, `compiler-ops`, `convert-lo`, `mask-bits`, `impl-defined` | | |
| | **B** | Instrumentation and enforcement | Build `specfill`, then make it authoritative | `specfill`, `fill-gen`, `named-results`, `gen-docs`, `fill-enforce` | | |
| | **C** | Conformance | Behavioral testing against `simdref` | `ci-audit`, `conform-fixed`, `validate-amd64`, `conform-scalable`, `conform-flip` | | |
| | **D** | Spec content | Writing spec functions, docs, and bodies — the shared bottleneck | `doc-explore`, `spec-common`, `spec-all` | | |
| | **E** | Retiring the old inputs | The five non-spec doc sources, then `categories.yaml` | `comments-yaml`, `doc-triage`, `handwritten-fill`, `template-docs`, `cat-rehome`, `compiler-ops-impl`, `cat-delete` | | |
| | **F** | simdgen internals | The signature refactor, then sourcing signatures from spec | `sig-refactor`, `sig-from-spec` | | |
| ## Dependencies | |
| A task is startable when every prerequisite is ticked. Hard dependencies are | |
| listed as `Needs`; soft dependencies — work is possible but wasteful or | |
| lower-value without them — as `Prefers`. | |
| Tasks are named, not numbered, so that a task's identity survives inserting | |
| another one next to it, and so that a CL saying "implements `cat-rehome`" still | |
| means something a year later. Ordering lives in the row order instead: the table | |
| is topologically sorted (both `Needs` and `Prefers` count as edges), with ties | |
| broken by track, so reading it top to bottom is a legal order to do the work in, | |
| and `cmd/spectasks` fails if an edit breaks that. Add a task by putting its row | |
| wherever its prerequisites and track allow. | |
| | ✓ | T | Task | What it is | Needs | Prefers | | |
| |---|---|---|---|---|---| | |
| | [x] | A | `width` | Resolve scalable width | — | — | | |
| | [ ] | A | `compiler-ops` | Compiler-only ops decision | — | — | | |
| | [ ] | A | `convert-lo` | `ExtendLo*`/`ConvertLo*` shape and naming | — | `width` | | |
| | [ ] | A | `mask-bits` | Mask bitmap load/store shape and naming | — | `width` | | |
| | [ ] | A | `impl-defined` | Implementation-defined behavior policy | — | — | | |
| | [x] | B | `specfill` | API walker + report mode | — | — | | |
| | [x] | B | `fill-gen` | Wire `specfill` into the generators | `specfill` | — | | |
| | [x] | B | `named-results` | Reconcile parameter and result names with spec | `fill-gen` | — | | |
| | [x] | B | `gen-docs` | Inject spec docs into generated output | `fill-gen` | `named-results` | | |
| | [x] | B | `fill-enforce` | Signature + name mismatch becomes an error | `named-results` | — | | |
| | [x] | C | `ci-audit` | CI/hardware audit | — | — | | |
| | [ ] | C | `conform-fixed` | Conformance harness, fixed-width types | `ci-audit` | `specfill` | | |
| | [ ] | C | `conform-scalable` | Conformance harness, scalable types | `width`, `conform-fixed` | — | | |
| | [x] | D | `doc-explore` | Doc-mechanism exploration on a hard family | — | — | | |
| | [x] | D | `spec-common` | Spec the common operations | `width`, `doc-explore` | `specfill` | | |
| | [ ] | C | `validate-amd64` | Validate spec bodies against amd64 | `conform-fixed`, `spec-common`, `impl-defined` | — | | |
| | [ ] | C | `conform-flip` | Flip conformance direction | `validate-amd64`, `conform-scalable` | — | | |
| | [ ] | D | `spec-all` | Bulk spec migration | `spec-common`, `convert-lo`, `mask-bits` | `specfill`, `fill-enforce` | | |
| | [ ] | E | `doc-triage` | Hand-written doc conflict triage | `specfill` | `spec-common` | | |
| | [x] | E | `comments-yaml` | Drop spec-covered docs from `midway/comments.yaml` | `gen-docs`, `spec-common` | `fill-enforce` | | |
| | [ ] | E | `handwritten-fill` | Enable `-w` on hand-written files | `gen-docs`, `doc-triage` | — | | |
| | [ ] | E | `template-docs` | Retire generator-supplied API docs | `gen-docs`, `spec-all` | — | | |
| | [ ] | E | `cat-rehome` | Re-home `categories.yaml` non-doc fields | `template-docs` | — | | |
| | [ ] | E | `compiler-ops-impl` | Implement compiler-only ops representation | `compiler-ops`, `cat-rehome` | — | | |
| | [ ] | F | `sig-refactor` | simdgen signature refactor | — | — | | |
| | [ ] | F | `sig-from-spec` | simdgen sources signatures from spec | `sig-refactor`, `cat-rehome`, `compiler-ops-impl` | — | | |
| | [ ] | E | `cat-delete` | Delete `categories.yaml` | `sig-from-spec` | — | | |
| **Critical path.** [SPEC-TASKS.dot](SPEC-TASKS.dot) draws it in red: every task | |
| on some longest chain of `Needs` edges. Slipping any of them slips the whole | |
| project. `cmd/spectasks` derives it from the table, so it is not restated here. | |
| Track B has no prerequisites, so it can start immediately and run to completion | |
| at its own pace — though `gen-docs` gates three of track E's tasks. | |
| `sig-refactor` is independent of everything until `sig-from-spec`. | |
| ## Milestones | |
| Milestones carry no work of their own; they are the points worth announcing. A | |
| milestone is reached when its tasks are ticked, so there is no separate state to | |
| maintain and nothing here to keep in sync. | |
| | Milestone | Reached after | What it means | | |
| |---|---|---| | |
| | **M1 — Enforced** | `fill-enforce` | Spec is authoritative for what it covers: a name or signature that drifts from it fails the build, in both packages, on all four targets. | | |
| | **M2 — Documented** | `comments-yaml`, `handwritten-fill`, `spec-all`, `template-docs` | Every operation's doc comment in both public packages derives from spec — SPEC-STATS.txt's "docs sourced from spec" reads 100%. | | |
| | **M3 — Gated** | `conform-flip` | Conformance runs spec-first, so an architecture added later must match spec before it can land. | | |
| | **M4 — Complete** | `cat-delete` | `categories.yaml` is gone and spec is the only hand-written description of the API; see the section below. | | |
| ## The graph | |
| [SPEC-TASKS.dot](SPEC-TASKS.dot) draws the two tables above; the rendered graph | |
| carries its own key. Track is node colour rather than a cluster, because a | |
| cluster has to be laid out contiguously and that fights the left-to-right | |
| dependency order the graph exists to show. | |
| dot -Tsvg SPEC-TASKS.dot -o SPEC-TASKS.svg | |
| It is generated, and its generator is also a checker. This document states its | |
| dependencies three times — Part 2's table, Part 3's `Needs:`/`Prefers:` lines, | |
| and Part 3's `Blocks:` lines, which are the same edges reversed — so | |
| `go test ./cmd/spectasks` fails if they disagree, or if the graph has not been | |
| regenerated since. It also derives the critical path. | |
| ## Start here | |
| The startable tasks are the ones with no unticked `Needs` in the dependency | |
| table. For a team of one to three, this is a reasonable opening allocation, | |
| covering every task with no `Needs` at all: | |
| - **Strongest on spec's type model:** `width`, then `convert-lo` and | |
| `mask-bits`. These root the longest chain and cost the most if deferred. | |
| - **Owns API semantics:** `impl-defined` and `compiler-ops` — pure | |
| decisions, no code, and everything downstream waits on them. | |
| - **Likes tooling:** `specfill`, then `fill-gen` and `named-results`. | |
| - **Working alone on something large:** `sig-refactor`. No prerequisites, | |
| verifiable byte-for-byte at every step. | |
| - **Cheap and worth doing early regardless:** `doc-explore` and `ci-audit`. | |
| ## What "done" looks like | |
| After `cat-delete`, `simd/internal/spec` is the only hand-written description of | |
| the API: every operation's name, signature, and documentation derives from it, | |
| in both public packages and on all four targets, and `specfill` fails the build | |
| if any of them drift. `categories.yaml` is gone. `midway/comments.yaml` | |
| documents only what spec does not describe: the vector types, `VectorSize`, | |
| `String` and `Len`. wasmgen and tmplgen no longer carry doc strings. | |
| Hand-written implementations remain hand-written, but their operation doc | |
| comments are generated. | |
| simdgen still exists and is still unification-based — §1.1 explains why. What | |
| changes is its job: the unifier no longer defines the API, it only expands one | |
| `go_GOARCH.yaml` entry into all the concrete machine instructions matching it. | |
| `go_GOARCH.yaml` stays, and so do its quirk fields, because instruction | |
| irregularity is real and has to live somewhere. The compiler backend generation | |
| is untouched throughout. | |
| --- | |
| # Part 3 — Task details | |
| Many completion criteria are "generated output is byte-identical", checked with | |
| `go run . -diff` from `_gen`. That runs every generator, including simdgen for | |
| all three of its targets, so it needs the XED and ARM ISA data first — see | |
| `fetch-xed.sh` and `fetch-arm64.sh`. There is no `-arch` flag on the driver; | |
| `-arch` belongs to `simdgen` when run directly. | |
| ## Track A — Decisions | |
| Human judgment calls. None has prerequisites, and each blocks downstream work — | |
| which is why they lead. Each is small in code and large in consequence. | |
| ### `width` — Resolve scalable width | |
| **Done:** [x] · **Needs:** — · **Blocks:** `spec-common`, `conform-scalable` · **Risk:** `risk-concrete-width` | |
| `simd/internal/spec/types.go` hardcodes `scalableWidth = 4096`, and `simdref` is | |
| built at that width, while `archsimd`'s SVE uses the runtime `vl()`. A | |
| conformance test of scalable types therefore compares vectors of different | |
| lengths. Spec's own TODO flags this and it is unresolved. | |
| 4096 bits in particular causes trouble for operations that index into vectors, | |
| such as shuffles, which is why every platform other than RISC-V RVV is 2048 bits | |
| or less. Resolving this task may require deciding whether we'll ever support | |
| vectors more than 2048 bits wide. | |
| Note what is *not* wrong: spec's type system was designed for scalable widths and | |
| handles them. The gap appears only when spec is used as a concrete conformance | |
| oracle, and the open question is narrow — whether spec can execute at a | |
| caller-chosen width without disturbing the generic type machinery. | |
| This is first because the fix may change spec's type model — making width a | |
| runtime parameter rather than a compile-time constant — which reaches every spec | |
| function body. Resolving it after a hundred bodies exist is far worse than | |
| resolving it now, while there are few (SPEC-STATS.txt §1). | |
| **Commits.** (1) Write up the options with their consequences for `Vec`, | |
| `makeVec`, `lanes`, `refgen`, and the conformance harness; seen so far: | |
| parameterize width at spec execution, make `simdref` width-configurable at | |
| construction, or scope conformance to fixed widths initially and defer. (2) | |
| Implement the chosen approach in `internal/spec`. (3) Regenerate | |
| `internal/simdref` and confirm `cmd/specls` output is unchanged, or explain what | |
| changed. | |
| **Done when** a scalable-typed operation can be executed through `simdref` at a | |
| width matching a chosen target, and the choice is documented in | |
| `internal/spec/doc.go`. | |
| ### `compiler-ops` — Compiler-only ops decision | |
| **Done:** [ ] · **Needs:** — · **Blocks:** `compiler-ops-impl` | |
| Decide how the compiler-only entries — those that produce no exported | |
| declaration (SPEC-STATS.txt §4) — and the `noTypes` / `noGenericOps` switches | |
| are represented once `categories.yaml` is gone. This is the only hard blocker in | |
| the chain without an obvious answer, so make it early. | |
| Note §1.7: because `specfill` is scoped to exported declarations, this does | |
| *not* block the checker or conformance tracks. | |
| The design note must also say where the following go once `categories.yaml` is | |
| gone: the compiler-only entries' `documentation:`, which feeds the doc comments | |
| of unexported methods in `archsimd/ops_internal_*.go`; `commutative`, `signed` | |
| and `out` on those entries; and the `go_arm64.yaml` use of `noGenericOps`. This | |
| documentation is strictly internal-facing, so it is held to a lower standard | |
| than user-facing API docs. It need not come from spec, go through `specfill`, or | |
| satisfy M2. Wherever it lands is acceptable as long as it is kept next to the | |
| compiler-only op's other description. | |
| **Commits.** One design note in this document or in `internal/spec/doc.go`, | |
| listing all compiler-only operations from `_gen/simdgen` (`grep -h "go: [a-z]" | |
| ops/*/categories.yaml | sort -u`; also the capitalized `noTypes` entries) and | |
| stating for each family where its description, documentation, and remaining | |
| fields live once `categories.yaml` is gone. Candidates: a `//specgen:unexported` | |
| directive; a fourth simdgen input holding compiler-only ops; or leaving them in | |
| `go_GOARCH.yaml` on the grounds that they are per-architecture lowering details | |
| rather than API. | |
| **Done when** the decision is written down, including documentation and all | |
| remaining fields, and `compiler-ops-impl` could be implemented from it without | |
| further design work. | |
| ### `convert-lo` — `ExtendLo*` / `ConvertLo*` shape and naming | |
| **Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects` | |
| Settle how the low-elements widening family is shaped and named, across both | |
| fixed-width and scalable vectors. An API design decision, not a mechanical fix. | |
| Three entangled problems: | |
| 1. **Shape.** `ExtendLoLToZ` declares `x Vec[E,W]` and `z Vec[zE,W]` — the same | |
| total width. The shipped API widens instead: `Int8x16.ExtendLo8ToInt64()` | |
| takes 128 bits and returns 512. Spec therefore misses 8 real declarations and | |
| invents width-preserving ones no platform has. | |
| 2. **Naming.** `//specgen:name ExtendLo{zL}To{zE}` embeds the *result lane | |
| count*, so the same conceptual operation gets a different name at each width. | |
| The API has settled on 128-bit receivers only, which hides this. | |
| 3. **Scalable.** The `converts.go` TODO already notes the literal `zL` cannot | |
| work for scalable vectors, and suggests dropping the number entirely since it | |
| is implied by the receiver and target types — e.g. `Uint8x16.ExtendLo2ToUint64` | |
| is really the low *eighth*, not the low half, so the number is arguably | |
| misleading even where it works. | |
| Any renaming here is API-visible and should be decided before `spec-all` | |
| migrates the Converts family, which is where the bulk of `categories.yaml`'s | |
| Converts documentation lands. | |
| **Commits.** (1) Write up the options and their consequences for fixed-width and | |
| scalable vectors, including whether the lane count stays in the name. (2) | |
| Re-shape `ExtendLoLToZ`, and re-check `ConvertLoLToZ`, whose `zL<xL` constraint | |
| currently admits conversions no hardware offers. (3) If the API name changes, | |
| land the rename through the generators as its own CL, separate from the spec | |
| change. | |
| **Done when** `cmd/specls` produces exactly the `ExtendLo*` declarations the API | |
| has, plus whatever maximalist extras are deliberate, and the scalable naming | |
| question is answered in `internal/spec/doc.go`. | |
| ### `mask-bits` — Mask bitmap load/store shape and naming | |
| **Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects` | |
| Settle the scalable counterpart of fixed-width `ToBits` and `MaskNxMFromBits` | |
| (e.g. `Mask8x16FromBits`), which today are `Mask*s.Store` and `LoadMask*s`. Also | |
| settle what `Store` and `ToBits` mean on portable `simd.Mask*s`, whose midway | |
| docs currently describe element semantics. | |
| The API declares `Store(bits []uint16)` only on SVE's scalable masks | |
| (`archsimd/types_sve.go`), paired with `LoadMask8s(bits []uint16)` etc. It | |
| stores SVE predicate bits at byte granularity. No fixed-width mask has `Store`; | |
| fixed-width masks have `ToBits() UintN` instead (spec: `MaskToBits`, | |
| `internal/spec/masks.go`, constrained to `FixedWidth`). So `Mask*s.Store` has a | |
| different meaning from every other `Store` — a candidate violation of the "one | |
| name, one meaning" rule. Its bit layout is SVE's representation, not a per-lane | |
| one, which arguably fails §1.3's "claim about the operation's own semantics" | |
| test. | |
| **Commits.** (1) Write up the options and their consequences for fixed-width and | |
| scalable masks, and for the portable package. (2) Change spec to match. (3) If | |
| the API name changes, land the rename through the generators as its own CL. | |
| **Done when** SPEC-STATS.txt §5 lists no mask `Store` defect, and the choice is | |
| recorded in `internal/spec/doc.go`. | |
| ### `impl-defined` — Implementation-defined behavior policy | |
| **Done:** [ ] · **Needs:** — · **Blocks:** `validate-amd64` · **Risk:** `risk-impl-defined-behavior` | |
| Decide what counts as implementation-defined behavior in the SIMD API, and how | |
| spec expresses it so conformance testing stays meaningful. Architectures | |
| legitimately differ on corners — the last bits of a float result, out-of-range | |
| shuffle indices, saturation edges, NaN propagation and signed zero. Spec's | |
| `math.go` already carries a TODO on `DotProductPairs` overflow that names the | |
| problem and floats an "architecture dependent" designated panic as one possible | |
| answer. | |
| Two halves, and both need answering: | |
| 1. **What is implementation-defined?** A per-operation API-semantics judgment. | |
| Costly in both directions: declare too much and conformance stops catching | |
| real bugs; declare too little and correct implementations fail against spec. | |
| 2. **How is it written down?** There are at least two major categories of | |
| implementation-defined behavior: exceptional-value cases (e.g., overflow | |
| behavior is fully undefined) and precision (all architectures roughly agree, | |
| but maybe not to the last digit). Spec bodies return exact values today, and | |
| there is no vocabulary for "any value here", "within N ulp", or "unspecified | |
| but deterministic per platform". Whatever is chosen must be understood by | |
| both `refgen` and the conformance harness, and it changes what a spec body | |
| looks like — which is why this wants settling before the bulk of them are | |
| written. The existing suite's `checkSlicesLogInput` already takes a per-call | |
| tolerance (§1.9), which is prior art for the numeric half; it says nothing | |
| about the "unspecified but deterministic per platform" half. | |
| **Commits.** (1) Survey the known cases — `DotProductPairs` overflow, float | |
| conversion rounding, shuffle index out of range, saturation edges, NaN/±0 | |
| handling — classifying each as specified, implementation-defined, or undefined. | |
| (2) Choose and document the spec vocabulary in `internal/spec/doc.go`. (3) | |
| Implement it far enough that one operation with a genuinely | |
| implementation-defined corner round-trips through `refgen` and the harness. | |
| **Done when** an operation with an architecture-dependent corner can be written | |
| in spec, generated into `simdref`, and conformance-tested without a false failure | |
| on any supported target. | |
| ## Track B — Instrumentation and enforcement | |
| Build `specfill`, then make it authoritative. Independent of tracks C, D and F; | |
| `gen-docs` gates three of track E. | |
| ### `specfill` — API walker + report mode | |
| **Done:** [x] · **Needs:** — · **Blocks:** `fill-gen`, `doc-triage` | |
| The instrument the rest of the plan reads from. | |
| `specfill` builds the whole of `specdoc.Fill` and `cmd/specfill` — the walk, the | |
| spec index, the comparison, and the rewrite of doc comments and parameter/result | |
| names. `fill-gen` wires it into generators with rewriting off; `named-results` | |
| and `gen-docs` then turn on name filling and doc injection there, and | |
| `handwritten-fill` turns `-w` on for hand-written files. | |
| **Interfaces.** `specdoc.Fill` is the library entry point — Go source in, Go | |
| source out: `func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`. | |
| Generators call it on their buffer before writing (§1.5). When verification | |
| checks fail, `Fill` returns a `*Report` as an error containing the findings | |
| enumerated below. Callers can recognize this error via `errors.As` if they | |
| choose to inspect or combine findings across files. Option fields: | |
| - **RejectUnknown** controls whether an exported declaration with no spec entry | |
| is an error (recorded in `Report.UnknownDecls`). Off during the transition; | |
| generators flip it on after `spec-all`, so that adding an operation without | |
| writing its spec function fails at generation time rather than silently escaping | |
| coverage. | |
| - **AllowDocRewrite** controls whether a declaration spec covers may already | |
| carry spec-owned paragraphs (§1.6). When false (default), each one that does | |
| is an error (recorded in `Report.UnexpectedDocs`); implementation notes and | |
| directives are always allowed. Once generators emit no API prose of their own | |
| (`template-docs`), this is their guard against silent doc loss | |
| (`risk-silent-doc-loss`): a generator paragraph missing its note prefix is an | |
| error rather than something `Fill` quietly replaces. Until then generators set | |
| it to true. Hand-written files carry prose by design, so `cmd/specfill` always | |
| sets it to true and relies on `-diff` and its `-w` guards instead. | |
| - **AllowNameMismatches** controls whether name mismatches are tolerated. When | |
| false (default), they are errors. Generators can set it to true during the | |
| transition, until `named-results` reconciles them. | |
| Two more fields select what `Fill` rewrites, for callers that want only part of | |
| it: **NoFillDoc** leaves doc comments alone, and **NoFillNames** leaves unnamed | |
| arguments unnamed (and so reports them as name mismatches). By default `Fill` | |
| does both. Finally, the **Filename** field labels positions in findings. | |
| **Names.** `Fill` fills and checks names as §1.8 describes: an argument the | |
| declaration leaves unnamed (or names `_`) takes spec's name, and a named | |
| argument whose name differs from spec's is a mismatch. It rewrites a | |
| declaration's names only when that declaration has no type or name mismatch, so | |
| a mismatch has to be fixed before the rest of its declaration's names are | |
| filled. | |
| `cmd/specfill` is a `gentools`-based command that applies `specdoc` to | |
| hand-written files: it walks the non-test `.go` files in `simd` and `archsimd`, | |
| skips generated ones (their generators call `Fill` themselves), calls `Fill` on | |
| each with `AllowDocRewrite`, and prints the combined report. `-diff` shows what | |
| a rewrite would change. With `-w`, rewrite in place; `-w` aborts without | |
| rewriting when the report has any findings, or if there are uncommitted changes, | |
| so its git diff is the complete story of what it did. The `-no-fill-doc` and | |
| `-no-fill-names` flags set the corresponding options. CI enforcement is a `go | |
| test` in this package — `fill-enforce` wires it — not a command invocation. | |
| **Report.** The report is the permanent enforcement layer — every | |
| item in it remains useful after the transition as the API continues to evolve, | |
| and is what `fill-enforce` runs in CI. Coverage and name relations are | |
| transition metrics that converge to their targets; they stay in | |
| `cmd/specstats` / SPEC-STATS.txt and are not part of this report. | |
| The report holds only errors; it does not preview rewrites. Every consumer of | |
| `specdoc` is a `gentools` program, `cmd/specfill` included, so `-diff` already | |
| shows exactly what a rewrite would change against the files on disk, including | |
| any text it would drop. That diff is how rewrites are reviewed, and a report that | |
| recomputed it would only be a second copy to keep in sync. | |
| The possible errors are: | |
| 1. **Type signature mismatches** — parameter or result types, or their | |
| counts, disagree between spec and the declaration the walker found. | |
| 2. **Name signature mismatches** — a named parameter or result whose name | |
| differs from spec's. Unnamed ones are filled rather than reported, unless | |
| `NoFillNames` is set. Omitted when `AllowNameMismatches` is set. | |
| 3. **Check (a) violations** — §1.6 ordering violations in declarations spec | |
| covers. `Fill` leaves such a comment as it is. Declarations spec does not | |
| cover are not checked: the ordering rule exists so rewrites are idempotent, | |
| and `Fill` never rewrites them. Hand-written files in particular may hold | |
| declarations spec will never cover. Each declaration is checked from the | |
| moment spec covers it. | |
| 4. **Unknown declarations** — exported declarations with no spec entry, when | |
| `RejectUnknown` is set. | |
| 5. **Unexpected doc comments** — declarations spec covers that carry | |
| spec-owned paragraphs, when `AllowDocRewrite` is false. | |
| **Commits.** (1) `specgen`: add an index over `Load()`'s result keyed by `(recv, | |
| name)`; add a function that classifies comment paragraphs as spec-owned or | |
| implementation note per §1.6; reject spec docs that break check (b). (2) New | |
| package `_gen/specdoc` and new command `_gen/cmd/specfill`: AST declaration | |
| walker, signature and comment ordering comparison, report findings implementing | |
| the `error` interface, and `AllowDocRewrite` / `AllowNameMismatches` options. | |
| Signature comparison translates the parsed `*ast.FuncDecl` into spec's own | |
| representation — its types into `specexpr` types, which suffices because no | |
| qualified types exist (§1.4) — and compares that structurally against the | |
| `specgen.Func`. (3) Rewrite engine in `specdoc.Fill` (both doc comments and | |
| signature names) and in-place rewriting (`-w`, `-no-fill-doc`, `-no-fill-names`) | |
| in `cmd/specfill`. | |
| **Done when** `specfill`, run on the current tree, writes nothing (without `-w`) | |
| and prints all report findings in position order. | |
| `SPEC-STATS.txt` is the sanity check for signatures: `cmd/specstats` computes | |
| the same type and name mismatches, counting names the way `Fill` does (§1.8), so | |
| `specfill`'s report should agree with it line for line on those. Treat any | |
| disagreement as something to explain before shipping. Coverage and name | |
| relations stay in `specstats` as transition metrics; once they reach their | |
| targets, `specstats` can be retired. | |
| ### `fill-gen` — Wire `specfill` into the generators | |
| **Done:** [x] · **Needs:** `specfill` · **Blocks:** `named-results`, `gen-docs` | |
| Each generator passes its buffer through `specfill` before writing, with all | |
| rewriting off for now: `NoFillNames` (`named-results` turns name filling on) and | |
| `NoFillDoc` (`gen-docs` turns doc injection on). Generators also set | |
| `AllowNameMismatches`, because names still disagree with spec until | |
| `named-results`, and `AllowDocRewrite`, because they still emit API prose of | |
| their own until `template-docs`. | |
| So this task only wires the generators up, and generated output does not | |
| change. What does change is that `Fill`'s remaining checks now run at | |
| generation time: a type mismatch or a check (a) violation in generated output | |
| fails generation. Neither can be tolerated by an option, and as of `specfill`'s | |
| completion there are none in generated output. | |
| **Commits.** (1) Add a narrow post-processing hook to `gentools` that does not | |
| make gentools depend on specgen (see §1.5). (2) Wire simdgen. (3) Wire tmplgen, | |
| wasmgen, midway. | |
| **Done when** all four generators route through `specfill`, and generated | |
| output is byte-identical to the previous tree. | |
| ### `named-results` — Reconcile parameter and result names with spec | |
| **Done:** [x] · **Needs:** `fill-gen` · **Blocks:** `fill-enforce` | |
| Every declaration spec covers carries spec's parameter and result names. See | |
| §1.8. Declarations spec does not cover are out of scope; they gain names when | |
| `spec-all` covers them. | |
| SPEC-STATS.txt §2 has the counts, split the way `specfill` treats them: | |
| - **Named arguments whose names disagree with spec**, in generated or | |
| hand-written code. Each is fixed in whichever of spec or the API has the worse | |
| name — spec, a generator's template or input, or a hand-written file; either | |
| way the diff touches only names, and doc text that mentions them. These come | |
| first: `Fill` fills no names in a declaration with a mismatch. | |
| - **Unnamed arguments**, which `Fill` fills from spec. In generated code, that | |
| means turning name filling on in the generators. In hand-written files, | |
| `specfill -w -no-fill-doc` fills them; `-w` refuses while the report has | |
| findings, so this also needs any check (a) violations in hand-written files | |
| cleared (they are also `doc-triage`'s). | |
| Cross-target parameter-name divergence (also SPEC-STATS.txt §2) is only fixed | |
| here for pairs spec covers. The rest reach zero as `spec-all` covers them, when | |
| each new spec function has to agree with every target at once. | |
| **Commits.** (1) Reconcile name mismatches, grouped by generator or operation | |
| family, each commit with its regenerated output. (2) Stop setting `NoFillNames` | |
| and `AllowNameMismatches` in the generators, with the regenerated output: name | |
| filling is on, and a name mismatch in generated output fails generation. (3) | |
| Fill unnamed arguments in hand-written files with `specfill -w -no-fill-doc`. | |
| **Done when** SPEC-STATS.txt §2's name mismatches and unnamed counts are zero, | |
| and the generators run without `NoFillNames` or `AllowNameMismatches`. | |
| ### `gen-docs` — Inject spec docs into generated output | |
| **Done:** [x] · **Needs:** `fill-gen` · **Prefers:** `named-results` · **Blocks:** `comments-yaml`, `handwritten-fill`, `template-docs` · **Risk:** `risk-silent-doc-loss` | |
| Turn on doc injection in the generators by dropping `NoFillDoc`. Generators keep | |
| setting `AllowDocRewrite` until `template-docs` removes their own prose. | |
| Prefer landing this after `named-results`: spec's docs refer to argument names | |
| (§1.8), so injecting them before the generators fill those names leaves docs | |
| that mention names the signature does not bind. | |
| Expect a real diff, not a no-op: spec already overlaps several hundred | |
| `(receiver, name)` pairs (SPEC-STATS.txt §1), and almost none of the existing | |
| doc text matches spec's (SPEC-STATS.txt §3). So this rewrites that many doc | |
| comments. Use `go run . -diff` first to size the change exactly. | |
| Before any generator's docs are injected, its implementation notes must follow | |
| §1.6, or the first rewrite silently drops them. Known gaps: some notes say `CPU | |
| Features:` or omit a colon; simdgen's performance caveats ("A non-constant value | |
| of shift may result in significantly worse performance", and the `SetElem` | |
| jump-table line) have no prefix, and the `SetElem` one sits directly above its | |
| `Asm:` line with no blank line between them. `specfill`'s check (a) violations | |
| and a `-diff` of the output with injection on are the list to work from. | |
| **Commits.** (1) Fix note prefixes in simdgen, tmplgen, wasmgen and midway | |
| templates — `Performance:` for the caveats, a blank line before every note — | |
| with the regenerated output; the diff should touch only note lines. (2) Turn on | |
| doc injection in simdgen. (3) Turn it on in tmplgen, wasmgen, midway. Commits | |
| (2) and (3) include their regenerated output; verify that output changes only | |
| for operations spec covers, and review every changed doc comment to make sure | |
| nothing was unintentionally dropped. (4) If `fill-enforce` is already done, drop | |
| `NoFillDoc: true` from `cmd/specfill.TestFill` for generated files to start | |
| enforcing documentation matches. | |
| **Done when** generated output has no check (a) violations, all four generators | |
| inject docs, nothing was dropped without review, and the only differences | |
| against the previous tree are doc comments for operations spec covers. | |
| ### `fill-enforce` — Signature + name mismatch becomes an error | |
| **Done:** [x] · **Needs:** `named-results` | |
| For operations spec covers, disagreement fails. Missing spec functions remain | |
| tolerated. Note this does **not** depend on `sig-refactor`: `specfill` compares | |
| emitted source, not simdgen's internal model. | |
| Generated output is already checked as it is generated: type mismatches fail | |
| generation from `fill-gen` on, and name mismatches from `named-results` on. | |
| That misses hand-written files, and generated files edited by hand without | |
| rerunning their generator. This task closes both with a `go test` in | |
| `cmd/specfill` that runs `Fill` over every non-test `.go` file in `simd` and | |
| `archsimd`, generated or not, and fails on any finding. It sets | |
| `AllowDocRewrite`, since those files carry spec-owned docs by design, whether | |
| injected or hand-written. | |
| **Commits.** One, adding the test. | |
| **Done when** a deliberately introduced signature or argument-name change in any | |
| generated or hand-written exported declaration fails the build. | |
| ## Track C — Conformance | |
| Behavioral testing against `simdref`. Parallel to everything else; shares only | |
| the spec-coverage bottleneck. | |
| ### `ci-audit` — CI/hardware audit | |
| **Done:** [x] · **Needs:** — · **Blocks:** `conform-fixed` · **Risk:** `risk-conformance-env` | |
| Determine what can actually execute tests for amd64, arm64, wasm, and SVE — | |
| natively or emulated. The conformance suite's value for other architectures | |
| depends on it. | |
| "Execute tests" means: build `simd` and `archsimd` for the target and run | |
| `go test` against real or emulated hardware, at a speed that tolerates a few | |
| thousand generated conformance cases. SVE needs SVE-capable hardware or QEMU, and | |
| wasm needs a runtime. | |
| **Done when** one document or issue records, per target, whether it can run | |
| today, on what, how fast, and what it would take if not — marking each runnable, | |
| runnable-with-work (with the work named), or not runnable, so `conform-fixed` | |
| knows which targets it is writing tests for. | |
| **Results** | |
| | amd64 | gotip-linux-amd64 | gotip-linux-amd64_avx512 | | |
| | ---------------- | ----------------- | ------------------------ | | |
| | AVX | ✔ | ✔ | | |
| | AVX2 | ✔ | ✔ | | |
| | AVX512 | ✘ | ✔ | | |
| | AVX512BITALG | ✘ | ✔ | | |
| | AVX512GFNI | ✘ | ✔ | | |
| | AVX512VAES | ✘ | ✔ | | |
| | AVX512VBMI | ✘ | ✔ | | |
| | AVX512VBMI2 | ✘ | ✔ | | |
| | AVX512VNNI | ✘ | ✔ | | |
| | AVX512VPCLMULQDQ | ✘ | ✔ | | |
| | AVX512VPOPCNTDQ | ✘ | ✔ | | |
| | AVXAES | ✔ | ✔ | | |
| | AVXPCLMULQDQ | ✔ | ✔ | | |
| | AVXVNNI | ✘ | ✔ | | |
| | FMA | ✔ | ✔ | | |
| | SHA | ✘ | ✔ | | |
| | VAES | ✘ | ✔ | | |
| | VPCLMULQDQ | ✘ | ✔ | | |
| For amd64, use the gotip-linux-amd64_avx512 gomote for testing. | |
| | arm64 | gotip-linux-arm64 | qemu-aarch64 -cpu neoverse-n2 | qemu-aarch64 -cpu neoverse-v1 | | |
| | --------------- | ----------------- | ----------------------------- | ----------------------------- | | |
| | PMULL | ✔ | ✔ | ✔ | | |
| | SVE | ✘ | ✔ | ✔ | | |
| | SVE2 | ✘ | ✔ | ✔ | | |
| | SVE vector size | n/a | 128 | 256 | | |
| `qemu-aarch64 -cpu neoverse-n1` (which matches the capabilities of | |
| gotip-linux-arm64) is roughly 7x slower than `gotip-linux-arm64`, so well within | |
| the range of practical conformance testing. | |
| For NEON, use the gotip-linux-arm64 gomote for testing. For SVE, use | |
| `qemu-aarch64 -cpu neoverse-v1`. | |
| | wasm | js-node | wasip1-wazero | | |
| | ------- | ------- | ------------- | | |
| | SIMD128 | ✔ | ✔ | | |
| `node` is about 3x faster than `wazero`. | |
| For wasm, use `node` for testing. | |
| ### `conform-fixed` — Conformance harness, fixed-width types | |
| **Done:** [ ] · **Needs:** `ci-audit` · **Blocks:** `validate-amd64`, `conform-scalable` · **Prefers:** `specfill` | |
| Generated tests comparing `archsimd` against `internal/simdref`. The generator's | |
| input is spec (hand-written source), so this is not a generator reading another | |
| generator's output. | |
| **Commits.** (1) Input generation: start from | |
| `archsimd/internal/simd_test/helpers_test.go`, which already has the per-type | |
| edge-case vectors, the mask patterns and the window-walking helpers (§1.9); | |
| decide whether to lift them into a package the generated tests can import. (2) | |
| Test generator producing one test per `(recv, name)` spec covers. (3) Wire into | |
| the SIMD test suite, skipping operations whose spec body panics. | |
| **Done when** every operation spec covers, on every target `ci-audit` marked | |
| runnable, is exercised against `simdref` with both edge-case and random inputs, | |
| and failures name the operation, the input, and both results. | |
| ### `conform-scalable` — Conformance harness, scalable types | |
| **Done:** [ ] · **Needs:** `width`, `conform-fixed` · **Blocks:** `conform-flip` | |
| Extend `conform-fixed`'s harness to the scalable types (`Int8s` … `Float64s`, | |
| `Mask8s` … `Mask64s`) on SVE. Separate from `conform-fixed` because it cannot be | |
| written until `width` settles how spec executes at a concrete width: the harness | |
| must construct `simdref` values at the width the hardware reports through `vl()` | |
| rather than at spec's fixed `scalableWidth`. | |
| **Commits.** (1) Teach input generation to size vectors from the runtime vector | |
| length. (2) Extend the test generator to scalable receivers. (3) Wire into the | |
| suite, gated on SVE availability per `ci-audit`. | |
| **Done when** scalable operations spec covers are conformance-tested on SVE at | |
| the hardware's actual vector length. | |
| ### `validate-amd64` — Validate spec bodies against amd64 | |
| **Done:** [ ] · **Needs:** `conform-fixed`, `spec-common`, `impl-defined` · **Blocks:** `conform-flip` | |
| Run conformance with amd64 as the oracle. Every failure is a spec bug until | |
| proven otherwise. See §1.9. | |
| **Commits.** One per batch of spec fixes; keep spec changes separate from harness | |
| changes so the reason for each is legible. | |
| **Done when** the harness passes on amd64 for every operation spec covers, with | |
| every remaining difference either fixed in spec or recorded as | |
| implementation-defined under `impl-defined`'s policy. | |
| ### `conform-flip` — Flip conformance direction | |
| **Done:** [ ] · **Needs:** `validate-amd64`, `conform-scalable` | |
| Spec becomes the oracle for every target, not just amd64. The second half is the | |
| contributor-facing note — likely in `internal/spec/doc.go` and the SIMD README — | |
| describing the workflow: add the operation to spec if it is not there, implement | |
| it, run conformance. | |
| **Done when** conformance runs and fails the build on all targets `ci-audit` | |
| marked runnable, and the workflow is written down where someone adding an | |
| architecture will find it. | |
| ## Track D — Spec content | |
| Writing spec functions, docs, and bodies. The shared bottleneck the other tracks | |
| draw on. | |
| ### `doc-explore` — Doc-mechanism exploration on a hard family | |
| **Done:** [x] · **Needs:** — · **Blocks:** `spec-common` · **Risk:** `risk-doc-template-ceiling` | |
| As of CL 831485, roughly one doc line in eight in `categories.yaml` carries | |
| shape-specific literals — `[x0, x1, x2, x3, ...]`, `128-bit`, `0b_11_01_00_10` — | |
| concentrated in `AddSub`'s `Concat*Pairs` and `Moves`' | |
| `concatSelectedConstant*`. Those docs cannot be produced by substituting `{vL}` | |
| into one template; each shape needs its own text. | |
| The expected mechanism is multiple spec functions sharing one `//specgen:name`, | |
| which specgen supports because `funcKey` is `(recv, name)` and the constraint | |
| sets are disjoint. **Verify that before bulk migration**, not after: discovering | |
| it does not scale once 150 easy docs are written means retrofitting all of them. | |
| **Commits.** (1) Migrate `ConcatAddPairs` / `ConcatSubPairs` into spec, docs | |
| included; their text differs between 16/32-bit and 64-bit elements. Verify via | |
| `cmd/specls`. (2) Migrate the `Grouped` variants, whose docs vary by element size | |
| *and* by the 128-bit grouping — two axes, and the real test of the mechanism. (3) | |
| `ShiftSaturated`, `ShiftAllRight`, `ShiftRight`, whose docs differ between signed | |
| and unsigned elements; `categories.yaml` selects the variant with a `signed:` | |
| join key, which this retires. (4) If the mechanism proves insufficient, extend it | |
| (conditional doc text is the likely shape) and document the extension in | |
| `internal/spec/doc.go`. | |
| `concatSelectedConstant*` is harder still — four spec functions for one API name | |
| — but it is *unexported*, so it sits outside `specfill`'s scope per §1.7 and is | |
| undecided until `compiler-ops`. Reach for it only if `compiler-ops` has landed | |
| and you want a sterner test. | |
| **Done when** `cmd/specls` reproduces the existing doc text for all three | |
| families, or the doc mechanism has been extended so that it can. | |
| ### `spec-common` — Spec the common operations | |
| **Done:** [x] · **Needs:** `width`, `doc-explore` · **Blocks:** `validate-amd64`, `comments-yaml`, `spec-all` · **Prefers:** `specfill` · **Risk:** `risk-ref-impl-effort` | |
| The method set `midway/comments.yaml` documents: its `.common_methods` block | |
| less `String` and `Len` — `Add`, `Sub`, `Min`, `Max`, `And`, `Or`, `MulAdd`, the | |
| compares, and the rest of the portable core. `SPEC-STATS.txt` tracks it as | |
| "portable core", which is this task's completion bar. Highest leverage in the | |
| plan — these are the easiest reference implementations to write correctly, they | |
| unlock `comments-yaml` (the worst doc divergence, in the widest-audience | |
| package), and they are exactly what a new architecture implements first. | |
| The file's keys are not all operations: besides the method docs, it documents | |
| vector *types*, functions and templates, and carries receiver names keyed to the | |
| shared block. Only the methods/functions are in scope here, and this excludes | |
| non-spec residue (`String`, etc, §1.7). | |
| **Commits.** Start by sampling ~10 operations at random and timing how long it | |
| takes to implement them, to calibrate `risk-ref-impl-effort` before committing | |
| to the rest. Then one commit per family (arithmetic, bitwise, min/max, compares, | |
| multiply-accumulate), each adding the spec functions with docs and bodies and | |
| confirming `cmd/specls` output. | |
| **Done when** every *method* name in `midway/comments.yaml` other than non-spec | |
| residue resolves to a spec function, with a real body, unblocking | |
| `comments-yaml`. | |
| ### `spec-all` — Bulk spec migration | |
| **Done:** [ ] · **Needs:** `spec-common`, `convert-lo`, `mask-bits` · **Prefers:** `specfill`, `fill-enforce` · **Blocks:** `template-docs` · **Risk:** `risk-ref-impl-effort` | |
| The remaining exported operations, prioritized by the names produced by more | |
| than one generator (SPEC-STATS.txt §1) — highest drift risk, and each migration | |
| fixes several files at once. Best-effort bodies are acceptable; `panic("not | |
| implemented")` is safe because conformance skips it. A *wrong* body is worse | |
| than no body. | |
| Two denominators, both useful. Against the API: the `(receiver, name)` pairs | |
| SPEC-STATS.txt §1 shows uncovered, which is what the scoreboard tracks. Against | |
| the input being retired: `categories.yaml`'s exported operation names, less the | |
| ones `spec-common` handles. They are different counts of different things — one | |
| `categories.yaml` name can expand to many API names — so quote whichever the | |
| audience needs and say which. | |
| **Why `convert-lo` and `mask-bits` gate this.** Only the Converts and LoadStore | |
| families actually depend on them — `convert-lo` may rename `ExtendLo*` / | |
| `ConvertLo*`, and `mask-bits` may rename the scalable mask bitmap pair, and | |
| renaming after the docs are migrated means doing them twice. If either is still | |
| open when this task starts, the other families are unaffected and are the place | |
| to begin. | |
| **Commits.** One per operation family, mirroring the `ops/*/` grouping. Each | |
| spec function gains `//specgen:commutative` where the operation is commutative; | |
| the directive already exists and is parsed by `specgen`. Drop any | |
| `comments.yaml` entries spec gains as they are covered. Include all regenerated | |
| outputs in each commit. Once `gen-docs` has landed, each family's regeneration | |
| rewrites that family's docs, so review the diff to ensure all doc comment | |
| content is equivalent: any dropped text is either something spec should say or | |
| an implementation note missing its prefix. | |
| **Done when** `specfill` reports no remaining API declarations uncovered by spec, | |
| other than the permanent residue named in §1.7 (`String`, `Len`). | |
| ## Track E — Retiring the old inputs | |
| Retiring the other documentation sources, then `categories.yaml` itself. | |
| ### `doc-triage` — Hand-written doc conflict triage | |
| **Done:** [ ] · **Needs:** `specfill` · **Prefers:** `spec-common` · **Blocks:** `handwritten-fill` · **Risk:** `risk-silent-doc-loss` | |
| Make it safe for `handwritten-fill` to rewrite hand-written files, by emptying | |
| the conflict set first. | |
| A conflict is a declaration where spec has a doc *and* the hand-written file has | |
| different spec-owned paragraphs (§1.6). Those are the only sites `-w` would | |
| overwrite: where spec has no entry, the declaration is left alone. Every | |
| conflict therefore resolves one of two ways — the prose says something spec | |
| should say (move it into spec), or it is about how a target implements the | |
| operation (make it an implementation note by giving it one of §1.6's prefixes, | |
| and `specfill` never touches it). Neither outcome loses text, which is what makes | |
| `-w` safe afterwards. | |
| Run `specfill -diff` over the hand-written files — every doc comment it would | |
| rewrite is a conflict — and sort each conflict into one of those two. | |
| SPEC-STATS.txt §3 breaks the conflict count down by file, with | |
| `simd_emulated.go` dominating and a long tail of `archsimd/slicepart_*.go`, | |
| `shuffles_amd64.go`, and `extra_amd64.go`. Some implementation text there has no | |
| prefix today — `extra_amd64.go`'s "This method compiles to VPTEST x, x" is a | |
| `Performance:` note — and the same pass fixes check (a) violations and note | |
| spellings in these files. | |
| The `slicepart_*.go` files matter most: they declare the `Load*Part` family spec | |
| *already* covers via `LoadZPart`, so their prose conflicts on day one — and one | |
| of them documents `LoadUint32x4Part` as loading "a Int32x4". | |
| **Done when** `specfill -no-fill-names -diff` is empty and `specfill` reports | |
| zero check (a) violations. | |
| ### `comments-yaml` — Drop spec-covered docs from `midway/comments.yaml` | |
| **Done:** [x] · **Needs:** `gen-docs`, `spec-common` · **Prefers:** `fill-enforce` | |
| Every method or function-template entry spec covers is dropped from | |
| `comments.yaml`. At the time of this task (after `spec-common`) that is the | |
| portable core less non-spec residue (`String`, etc, §1.7), plus the | |
| Load/LoadPart/Broadcast templates. Entries spec gains afterwards (in `spec-all`) | |
| are dropped when spec gains them. | |
| **Commits.** (1) Confirm spec covers the portable core; fill gaps. (2) Delete | |
| the covered entries from `comments.yaml`. (3) Run `specfill -w` over | |
| `simd/simd_emulated.go` — this overlaps `handwritten-fill`; sequence whichever | |
| lands first. | |
| **Done when** `comments.yaml` has no entry for any operation spec covers, midway | |
| sources those docs through `specfill`, and `simd_stubs.go` and | |
| `simd_emulated.go` carry identical spec-derived docs for them. | |
| ### `handwritten-fill` — Enable `-w` on hand-written files | |
| **Done:** [ ] · **Needs:** `gen-docs`, `doc-triage` · **Risk:** `risk-silent-doc-loss` | |
| Turn on doc rewriting for hand-written files, once `doc-triage` has emptied the | |
| conflict set — at which point `-w` rewrites only docs that already match what it | |
| would write. | |
| Two guards, both on `cmd/specfill` only. This does **not** change `gentools`' | |
| shared `-w`, which every generator uses: | |
| - `specfill -w` refuses to run when the report has any findings, or if there are | |
| uncommitted changes, so its diff is the complete story of what it did. | |
| - `specfill -w` never runs in CI; CI runs report mode (`fill-enforce`). | |
| **Done when** `specfill -w` over the hand-written files is a no-op on a clean | |
| tree. | |
| ### `template-docs` — Retire generator-supplied API docs | |
| **Done:** [ ] · **Needs:** `gen-docs`, `spec-all` · **Blocks:** `cat-rehome` | |
| Delete the doc strings embedded in wasmgen's and tmplgen's templates, and the | |
| `documentation:` fields on exported `categories.yaml` entries, leaving | |
| `specfill` to supply them. Mechanical once `spec-all` has covered the operations | |
| those generators emit — but not automatic, because their templates interleave | |
| prose with the implementation notes that stay with the generator (§1.6). | |
| **Commits.** One per generator, stripping the prose and keeping the notes; one | |
| for the `documentation:` fields on exported `categories.yaml` entries. Each | |
| includes its regenerated output. Then stop setting `AllowDocRewrite` in the | |
| generators, so a generator that emits API prose again fails generation. | |
| **Done when** no generator's source contains API prose, no exported | |
| `categories.yaml` entry has `documentation:`, regeneration is byte-identical | |
| to the previous tree, and the generators run without `AllowDocRewrite`. | |
| ### `cat-rehome` — Re-home `categories.yaml` non-doc fields | |
| **Done:** [ ] · **Needs:** `template-docs` · **Blocks:** `compiler-ops-impl`, `sig-from-spec` · **Risk:** `risk-cat-unknowns` | |
| Empty the exported entries of `categories.yaml` of everything but `go:` and | |
| `commutative:`, by sending each remaining field where §1.3's inventory says it | |
| belongs. `template-docs` has already removed `documentation:`; this is the rest. | |
| Compiler-only entries are untouched, for `compiler-ops-impl`. | |
| Expect surprises. §1.3's audit was static, and the unifier's behavior on field | |
| combinations was not exhaustively traced — `regexpTag`'s role is still | |
| unverified (§1.3). Mitigating factor: `go run . -diff` makes every step | |
| verifiable byte-for-byte against checked-in generated output. | |
| **Commits.** Follow §1.3's disposition column, one field per commit, confirming | |
| generated output is unchanged at each step: (1) delete `regexpTag` and `signed` | |
| from `categories.yaml` and from `go_*.yaml`, expecting most `go_*.yaml` uses to | |
| be inert already; (2) move `constImm` into `go_amd64.yaml`; (3) convert the | |
| `out`/`elemBits`/`bits` entries into spec function splits, which `spec-all` | |
| should already have done — verify. | |
| **Done when** every exported entry in `categories.yaml` has only `go:` and | |
| `commutative:` keys; compiler-only entries are untouched, for | |
| `compiler-ops-impl`. Generated output is byte-identical throughout. | |
| ### `compiler-ops-impl` — Implement compiler-only ops representation | |
| **Done:** [ ] · **Needs:** `compiler-ops`, `cat-rehome` · **Blocks:** `sig-from-spec` | |
| Implement whatever `compiler-ops` decided, for the compiler-only entries — | |
| those that produce no exported declaration — and the `noTypes` / `noGenericOps` | |
| switches. No design work should remain at this point; if it does, `compiler-ops` | |
| was not finished. | |
| These operations have no API surface — they exist only as SSA ops and lowering | |
| rules — so nothing here touches `specfill`, spec, or the public packages. It is | |
| entirely about giving simdgen somewhere other than `categories.yaml` to read them | |
| from. | |
| **Done when** no compiler-only entries remain in `categories.yaml` | |
| (SPEC-STATS.txt §4) and generated output is byte-identical. | |
| ### `cat-delete` — Delete `categories.yaml` | |
| **Done:** [ ] · **Needs:** `sig-from-spec` | |
| Delete `ops/*/categories.yaml`, the `!import` stub that loads it, and its | |
| handling in `simdgen/main.go`. By this point every field it carried has been | |
| re-homed (`cat-rehome`), its compiler-only entries have moved | |
| (`compiler-ops-impl`), and its signatures come from spec (`sig-from-spec`), so | |
| this should be pure deletion. | |
| **Done when** `categories.yaml` no longer exists, simdgen runs without it, and | |
| generated output is byte-identical. | |
| ## Track F — simdgen internals | |
| The signature refactor and, eventually, sourcing signatures from spec. | |
| ### `sig-refactor` — simdgen signature refactor | |
| **Done:** [ ] · **Needs:** — · **Blocks:** `sig-from-spec` · **Risk:** `risk-sig-refactor-convergence` | |
| Make the Go signature a first-class value on `Operation` instead of an emergent | |
| property of `shape()` → `classifyOp()` → a template name. Ideally shaped so it is | |
| directly comparable to `specgen.Func`'s `Recv`/`In`/`Out`. | |
| This collapses the stub templates (§1.1 counts them) to roughly three, because | |
| operand permutation and parameter naming become data rather than template | |
| identity; the `_21`, `_231Type1`, `_31` suffixes become an explicit permutation. | |
| It is also the empirical test of whether the unification approach can be kept. | |
| The refactor assumes the signature is expressible as data without a new pile of | |
| special cases; the quirk fields and `simdgen only supports…` panics catalogued | |
| in §1.1 are mostly *upstream* of the signature (XED inconsistency) or | |
| *downstream* (register shapes, rules), so the seam looks thin — but that is a | |
| prior, not a measurement. If it does not converge, that is the evidence for | |
| revisiting §1.1, earned cheaply on the few hundred lines §1.1's layer table | |
| isolates. | |
| **Commits.** (1) Introduce the signature type and a `GoSignature()` method | |
| covering the simplest classes (`op1`, `op2`, `op3`). (2) Extend to immediate | |
| variants (`op*Imm*`), which may be fiddly. (3) Extend to permuted and | |
| `VecAsScalar` variants. (4) Rewrite the stub templates to consume the signature | |
| value; delete the redundant templates. | |
| **Done when** `go run . -diff` shows byte-identical output at every commit. | |
| ### `sig-from-spec` — simdgen sources signatures from spec | |
| **Done:** [ ] · **Needs:** `sig-refactor`, `cat-rehome`, `compiler-ops-impl` · **Blocks:** `cat-delete` | |
| simdgen looks up the spec function by `(recv, name)` and asserts signature | |
| equality, reporting a structured diff on mismatch. **Key the lookup on | |
| `(recv, name)` only** — spec's own uniqueness key — and assert the signature | |
| separately. Keying on the signature makes a divergence indistinguishable from | |
| "not migrated yet", which is the failure this whole effort exists to catch. | |
| This is where `sig-refactor` pays off: the lookup compares against the signature | |
| value that refactor introduced, rather than against a rendered template. | |
| **Commits.** (1) Look up the spec function and assert equality, failing the | |
| generator run on mismatch. (2) Delete whatever signature information in | |
| `go_GOARCH.yaml` is now redundant. (3) Switch simdgen to reading commutativity | |
| from spec's `//specgen:commutative` directive and delete `commutative:` from | |
| exported `categories.yaml` entries. | |
| **Done when** simdgen derives every exported signature it emits from spec, | |
| commutativity comes from spec, and generated output is byte-identical. | |
| --- | |
| # Part 4 — Risks and unknowns | |
| Every risk is owned by a task, and its substance is stated there. This is the | |
| index. `risk-concrete-width`, `risk-impl-defined-behavior` and | |
| `risk-ref-impl-effort` are the expensive ones: each reaches every spec body, so | |
| each is far cheaper to settle before the bodies are written than after. | |
| | | Risk | What it is | Owner | | |
| |---|---|---|---| | |
| | **[ ]** | `risk-spec-defects` | Known spec defects | `convert-lo`, `mask-bits` | | |
| | **[x]** | `risk-concrete-width` | Executing spec at a concrete width | `width` | | |
| | **[ ]** | `risk-impl-defined-behavior` | Implementation-defined behavior | `impl-defined` | | |
| | **[x]** | `risk-doc-template-ceiling` | Doc-template ceiling — the fallback is a mechanism change affecting every doc written so far | `doc-explore` | | |
| | **[ ]** | `risk-ref-impl-effort` | Reference-implementation effort — every migrated operation needs a body that is *correct*, because conformance tests against it; the distribution of difficulty is unknown | `spec-common`, `spec-all` | | |
| | **[x]** | `risk-conformance-env` | Conformance execution environment — can CI run arm64/SVE/wasm, and at what cost? | `ci-audit` | | |
| | **[ ]** | `risk-sig-refactor-convergence` | `sig-refactor` may not converge — if it does not, §1.1 needs revisiting | `sig-refactor` | | |
| | **[ ]** | `risk-silent-doc-loss` | Silent doc loss — under §1.6, text without a note prefix is spec-owned, so implementation text nobody prefixed is dropped on the first rewrite; the guards are review of every rewrite's `-diff` and, once generators emit no prose of their own, `AllowDocRewrite` | `gen-docs`, `doc-triage`, `handwritten-fill` | | |
| | **[ ]** | `risk-cat-unknowns` | Unknown unknowns in `categories.yaml` — the field audit was static | `cat-rehome` | | |
| --- | |
| # Appendix A — Regenerating the two generated files | |
| ## `SPEC-STATS.txt` | |
| `cmd/specstats` generates [SPEC-STATS.txt](SPEC-STATS.txt), the transition | |
| scoreboard: | |
| ```sh | |
| cd $GOROOT/src/simd/archsimd/_gen | |
| go run . -w # (if necessary) ensure generator outputs are up-to-date | |
| go run ./cmd/specstats # print the report | |
| go run ./cmd/specstats -diff # see what's changed | |
| go run ./cmd/specstats -w # rewrite SPEC-STATS.txt in place | |
| ``` | |
| It loads `simd` and `archsimd` through `go/packages` for each build | |
| configuration, taking syntax only — no type-checking, per §1.4 — and reads | |
| `simd/internal/spec` through `specgen.Load`. It needs a recent Go toolchain and | |
| up-to-date output from the SIMD generators, but does not itself require any ISA | |
| data. Commit the regenerated file alongside whatever change moved the numbers; | |
| the diff is the record of what moved. | |
| It reports only figures that measure how far along the transition is: every one | |
| is a fraction heading for 100% or a count heading for zero, tagged with the | |
| tasks that move it; indented, untagged numbers are context breaking down the | |
| figure above them. If you add a figure, give it a denominator and a direction, | |
| or leave it out. | |
| One thing it deliberately does not do: it classifies divergence (§5) but fails | |
| on nothing; enforcement is `specfill`'s job. `specfill` should absorb the | |
| comparison half of this program rather than leave two implementations to drift. | |
| For one-off questions the report does not answer, `cmd/specls` prints the API | |
| spec currently expands to: | |
| ```sh | |
| go run ./cmd/specls # the whole expansion | |
| go run ./cmd/specls -f Add # one spec function | |
| go run ./cmd/specls -f Add -trace # why it expands the way it does | |
| ``` | |
| ## `SPEC-TASKS.dot` | |
| The task graph is drawn from Part 2's two tables by `cmd/spectasks`: | |
| ```sh | |
| go run ./cmd/spectasks # print the graph | |
| go run ./cmd/spectasks -w # rewrite SPEC-TASKS.dot in place | |
| dot -Tsvg SPEC-TASKS.dot -o /tmp/tasks.svg # render to SVG | |
| go run ./cmd/spectasks | grep -v ^-- | xdot - # view the current graph | |
| ``` | |
| It reads nothing but this file. Two things it does beyond drawing: | |
| It **checks**, from `go test ./cmd/spectasks`. Dependencies are stated three | |
| times here — Part 2's table, Part 3's `Needs:`/`Prefers:` lines, and Part 4's | |
| `Blocks:` lines, which are the same edges reversed — and the test fails if the | |
| three disagree, if the table is not topologically sorted, if a task is in no | |
| track or in two, or if an edge points at a task that does not exist. A second | |
| test fails if SPEC-TASKS.dot no longer matches the document, so an edit that | |
| skips the regeneration step is caught too. The command itself refuses to write a | |
| graph from an inconsistent document, but the test is where the problems are | |
| reported. | |
| It **derives the critical path** rather than restating one: a task is on it when | |
| its longest chain in plus its longest chain out spans the whole graph. That | |
| keeps every longest chain, not one picked by hand, so parallel roots of equal | |
| depth are all marked. |