| # Transitioning the Go SIMD generators to `internal/spec` |
| |
| This plan lays out how to transition the myriad generators and hand-written |
| sources of the SIMD API from disjoint, manually-enforced consistency, to a |
| single underlying executable API specification that provides all method |
| signatures, documentation, and checkable semantics. |
| |
| See [README.md](README.md) for background on the SIMD packages, the "one name, |
| one meaning" rule, the generators, how simdgen works, what spec and specgen are, |
| and how to run things. |
| |
| ## Table of contents |
| |
| - Part 1: Decisions and rationale |
| - Part 2: Task graph |
| - Part 3: Task details |
| - Part 4: Risks and unknowns |
| - Appendix A: Regenerating the two generated files |
| |
| ## The plan in brief |
| |
| `internal/spec` — generic Go functions describing each operation — becomes the |
| single hand-written description of the SIMD API. Today the same operation is |
| described in six places, and they have already drifted from each other (§1.5). |
| |
| Three things have to happen, and they are largely independent: |
| |
| 1. **Build the `specfill` checker.** One AST pass that injects spec's docs into |
| generated and hand-written source and verifies signatures against spec. Types |
| can be enforced almost at once — SPEC-STATS.txt §2 shows zero type-level |
| mismatches between spec and the shipped API. |
| 2. **Write the spec functions** (`spec-common` and `spec-all`). Coverage is in |
| SPEC-STATS.txt §1. This is the bottleneck the other |
| tracks draw on. |
| 3. **Retire the old inputs**: the other doc sources, then `categories.yaml`, at |
| which point the unifier no longer defines the API — it only expands one arch |
| entry into the machine instructions matching it. |
| |
| Behavioral conformance against `internal/simdref` (track C) runs in parallel. |
| Beyond its own `ci-audit`, it depends only on spec coverage and on the `width` |
| and `impl-defined` decisions. |
| |
| ## Monitoring plan progress |
| |
| Progress on this transition is tracked in two ways: |
| |
| **Task completion** is tracked directly in this document via the `[ ]` |
| checkboxes on each task and risk. The task table in part 2 summarizes all task |
| completion in one place. |
| |
| [SPEC-TASKS.dot](SPEC-TASKS.dot) shows a visual representation of the task graph |
| and completed tasks. It is generated directly from this document by running `go |
| run ./cmd/spectasks -w` from `_gen`. This generator also checks the consistency |
| of this doc, and the test in `./cmd/spectasks` enforces doc consistency in CI. |
| Rerun the generator after modifying any task metadata. |
| |
| **Measurements** live in [SPEC-STATS.txt](SPEC-STATS.txt), which tracks spec |
| coverage of the API, signature and name agreement, documentation sources, and |
| retirement of the old inputs. It is generated by running `go run ./cmd/specstats |
| -w` from `_gen`. Every figure in that file is a fraction heading for 100% or a |
| count heading for zero, tagged with the tasks that move it. Indented, untagged |
| numbers are context breaking down the figure above them. Part 1 of this doc |
| quotes measurements that motivated specific decisions, stamped with the CL at |
| which they were measured; those are frozen and do not need refreshing. |
| |
| ## How to implement a task |
| |
| Part 2 tells you which task to pick up: the next one whose prerequisites are all |
| ticked. [SPEC-TASKS.dot](SPEC-TASKS.dot) draws the same thing if a picture |
| helps. The commit sequences in Part 3 are guesses made before the work started; |
| deviate when the code tells you to, but say so in the CL description. |
| |
| Tasks and risks carry `[ ]` checkboxes. When you resolve one, tick the box, |
| update the task text to reflect the reality of the implementation so it remains |
| useful to reference when performing follow-on tasks (you can drop or summarize |
| the exact commit sequence), and regenerate [SPEC-TASKS.dot](SPEC-TASKS.dot). |
| Commit this as its own CL on top of the CLs that implement the task, and include |
| justifications and reasoning for any changes to the task text in the commit |
| message. |
| |
| ## Key concepts and terms |
| |
| **`specfill`** is the tool this plan builds first and then leans on. It is one |
| AST pass over Go source that injects spec-derived doc comments and parameter |
| and result names, and verifies signatures against spec. It exists as both a |
| package that can be integrated into generators and a standalone tool: |
| |
| - **`specdoc.Fill`** is the library entry point — Go source in, Go source out: |
| `func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`. |
| If verification checks fail, it returns a `*Report` as an error. Generators |
| call it on their `gentools` buffer before writing. |
| - **`cmd/specfill`** is the `gentools`-based command wrapping it, for running |
| over hand-written files. Where this document writes `specfill -w`, that is |
| the command; where it writes `specfill`, that is the library function. CI |
| enforcement is a `go test` in this package. |
| |
| These package and command names are suggestions; the split between library and |
| command is not. |
| |
| **Target** means one of amd64, arm64 (NEON), arm64 SVE, or wasm. See README.md. |
| |
| **Spec operation** means one function in package `spec`. Each expands across |
| element types, widths, and templated names into many API names and many |
| `(receiver, name)` pairs, so it is the grain spec is *authored* at — and the |
| only grain at which "implemented nowhere" means anything (SPEC-STATS.txt §5). |
| |
| **Operation** (unqualified) means the API-level operation a name denotes, |
| whichever of spec, `categories.yaml`, or generated source describes it. A |
| `categories.yaml` entry is called an *entry*; one operation name can have several. |
| |
| **Granularity.** Spec coverage is measured at three granularities: |
| `(target, receiver, name)` — every exported declaration, counted once per build |
| configuration; `(receiver, name)` — collapsed across targets, the primary |
| denominator since spec describes an operation once; and `name` — collapsed across |
| receivers too, coarser and misleading when a name exists on different receiver |
| sets in spec and the API (SPEC-STATS.txt §5). SPEC-STATS.txt §1 reports all |
| three. `String` and `Len` are excluded throughout (§1.7). |
| |
| --- |
| |
| # Part 1 — Decisions and rationale |
| |
| These are the load-bearing decisions. Each was contested during design. The |
| reasoning is recorded so it need not be re-litigated, while also making it |
| easier to revisit decisions if new evidence arises. |
| |
| Measurements in this section were taken at CL 831485 unless otherwise noted. |
| They should NOT be updated unless decisions need revisiting. |
| |
| ## 1.1 We are not rewriting simdgen |
| |
| The temptation is real: simdgen has 28 stub templates whose *names* encode |
| arity, operand permutation, and immediate placement (`op3_231Type1`, `op4_31`, |
| `op2Imm8_II`); 42 fields across `RawOperation` (23) and `Operand` (19); and 27 |
| bail-outs whose messages begin "simdgen only supports" or "simdgen does not". |
| But the layer accounting says the mess is small and the value is large (at |
| CL 831485; Go counts include `_test.go`; "signature derivation + emission" is |
| `simdgen/gen_*.go`, `godefs.go`, `arch.go`, and `types/operation.go`): |
| |
| | Layer | Lines | Affected by "was unification wrong?" | |
| |---|---|---| |
| | Platform loaders | 5,100 | No — irreplaceable | |
| | Unifier engine | 3,169 | Partly | |
| | Signature derivation + emission | 4,715 | The ~500 messiest lines, yes | |
| | `categories.yaml` | 1,284 | Deleted by this plan anyway | |
| | `go_GOARCH.yaml` | 4,475 | No | |
| |
| The part that would be thrown away if unification were the wrong call is the ~500 |
| messiest lines of signature derivation, against irreplaceable platform loaders |
| several times that size. |
| |
| A few thousand lines of hand-written YAML drive tens of thousands of lines of |
| generated API and compiler backend across four targets. That ratio is good. |
| What is *not* earning its keep is unification's role as **API definer** — the |
| `categories.yaml` ↔ `go_GOARCH.yaml` join — and this transition removes that role |
| by construction. The unifier's remaining job, expanding one arch entry into every |
| matching concrete instruction variant against the ISA database, it does well. |
| |
| ## 1.2 Spec is not a unifier input |
| |
| Feeding spec into unification via `DefBuilder` is easy and wrong. Spec's |
| structure would be flattened into the unstructured value language the moment it |
| crossed the boundary, recreating the exact failure mode we are escaping. Instead, |
| simdgen consumes `[]*specgen.Func` as a typed Go table. |
| |
| Furthermore, spec is inherently only aware of the structure of the Go SIMD API, |
| but the inputs to the unifier are in platform terms. Much of the output of the |
| unifier is information on how to map the platform structure into the Go SIMD |
| API, so we can't bring spec in until *after* that mapping has been processed. |
| |
| ## 1.3 What belongs in spec |
| |
| The test is: **is it a checkable claim about the spec's own semantics?** |
| |
| - `commutative` — yes. `Add(x,y) == Add(y,x)` is testable against the executable |
| body, and spec is the only place that *can* check it. |
| - `noGenericOps`, `noTypes` — no. Claims about generator output. |
| - `constImm` — no. A claim about x86 encoding. |
| - `regexpTag`, `signed` — no. Claims about the unifier. |
| |
| The test is self-policing: anything that passes it falls out of the conformance |
| testing we want anyway. Note it admits facts the *compiler* consumes — the wrong |
| line to draw is "API vs compiler"; the right one is "intrinsic to the operation |
| vs incidental to the mechanism or platform." |
| |
| For `commutative`, no name in `categories.yaml` currently needs |
| platform-specific commutativity, so a function-level directive in spec is |
| sufficient today. However, there may be corner cases we haven't uncovered; |
| notably, we have evidence that `Min`/`Max` on floats may not be commutative on |
| all platforms. It may suffice to simply not mark these operations as |
| commutative, and to document when an operation is not commutative for surprising |
| reasons. |
| |
| Applying the test to `categories.yaml`'s fields (at CL 831485; 217 entries, 195 |
| unique operation names, 163 exported): |
| |
| | Field | Count | Transition plan | |
| |---|---|---| |
| | `go` | 217 | → spec function name | |
| | `commutative` | 217 | → `//specgen:commutative` directive; `categories.yaml` copy removed by `sig-from-spec` | |
| | `documentation` | 211 | → spec doc comment; removed by `template-docs` | |
| | `regexpTag` | 25 | unification join key; role unverified (see `risk-cat-unknowns`) | |
| | `constImm` | 22 | → sideways into `go_amd64.yaml` (x86 encodings) | |
| | `out` | 14 | join key used only for selecting a doc variant; → multiple spec funcs sharing one `//specgen:name` | |
| | `elemBits` | 14 | same | |
| | `bits` | 4 | same | |
| | `signed` | 12 | same | |
| | `noGenericOps` 13 / `noTypes` 6 | 13 entries | **open question** — compiler-only ops | |
| |
| Task `cat-rehome` tackles most of the "transition plan" column. |
| `compiler-ops-impl` handles the `noGenericOps` / `noTypes` entries. |
| |
| ## 1.4 `specfill` parses; it does not type-check |
| |
| `ops_amd64.go` and `types_amd64.go` have minimal imports, and no exported |
| signature in either package uses a qualified type, so `go/parser` is sufficient — |
| no `go/types`, no `packages.Load`, no GOEXPERIMENT wrangling, no cross-module |
| loading of GOROOT std packages. |
| |
| This is better than a workaround. Type-checking sees one build configuration at a |
| time and would make the cross-architecture comparison *harder*. Parsing all files |
| regardless of build tag yields `(Int32x4) Add` from `ops_amd64.go`, |
| `ops_arm64.go`, and `ops_wasm.go` in a single pass, which is precisely the |
| comparison we want. `apisum` already does this, so it is proven on this API. |
| |
| `cmd/specstats` (Appendix A) takes a different route to the same end: it loads |
| syntax through `go/packages` once per build configuration, because it reports |
| per-target figures. It still does not type-check, and `specfill` does not need |
| per-configuration loading. |
| |
| ## 1.5 Two-pass doc injection, and why the "disadvantage" is the point |
| |
| The motivation is doc divergence. At CL 831485 there were **six independent |
| documentation sources** — `categories.yaml`, wasmgen inline strings, tmplgen |
| inline strings, `midway/comments.yaml`, hand-written `simd_emulated.go`, and |
| hand-written `archsimd` files — and none agreed with any other. All 90 |
| `(receiver, name)` pairs documented in both `simd` and `archsimd` had different |
| doc text, and within the portable package, `simd_stubs.go` (midway-generated) and |
| `simd_emulated.go` (hand-written) had drifted on 40 of 336 mirrored methods. |
| `Add` shipped with three independent wordings: |
| |
| | Where | Text | Source | |
| |---|---|---| |
| | `ops_amd64.go`, `ops_sve.go` | *Add adds corresponding elements of two vectors.* | `categories.yaml` | |
| | `ops_wasm.go` | *Add returns the result of adding x and y, elementwise.* | wasmgen, inline | |
| | `simd_stubs.go`, `simd_emulated.go` | *Add returns the element-wise sum of x and y.* | `midway/comments.yaml` | |
| |
| 124 method names are produced by more than one generator — the easiest place for |
| disagreement to creep in. simdgen accounts for the majority of declarations, but |
| tmplgen, wasmgen, midway, and hand-written code own the rest, and those sets |
| overlap — any instrument that only sees simdgen misses a substantial fraction of |
| the problem (SPEC-STATS.txt §1 has the per-generator counts). |
| |
| Generators pass their generated code to `specfill`, which checks signatures and |
| injects docs before writing to disk; in the end state, generators emit |
| declarations and implementation notes (§1.6) but no API prose of their own. The |
| alternative — having specgen emit the whole declaration, as `refgen` does — was |
| rejected because it would make agreement with spec true by construction and |
| leave nothing to verify. A generator whose internal model of the API had drifted |
| would be silently papered over. Because the template still produces the |
| signature, `specfill` sees two independent witnesses to the same declaration and |
| can compare them: docs in, signatures verified, coverage reported, one AST walk. |
| |
| Consequences: |
| |
| - **`specfill` is an override, not a replacement.** Generators keep emitting |
| their current docs; `specfill` replaces the spec-owned paragraphs only when spec |
| knows the operation. Fallback is therefore automatic and no flag day is |
| required. Prose-less emission is the *end state* reached when a generator's doc |
| source empties, not a precondition. |
| - **wasmgen and tmplgen need only a call before flush**, plus making the notes |
| they emit follow §1.6's prefixes. |
| - **Do not hook this into `gentools.Files.Flush()`.** `gentools` is a general |
| utility; coupling it to specgen would make every consumer depend on |
| `golang.org/x/tools/go/packages` and a loadable spec tree. A narrow hook |
| mechanism in gentools plus a few lines per generator keeps the boundary. |
| |
| ## 1.6 The doc-ownership convention |
| |
| `specfill` rewrites part of an existing doc comment, so it needs a rule for which |
| part it may touch, and running it twice must be idempotent. The obvious rule, |
| "everything above the first bare `//`", doesn't work. At CL 831485 only 2,759 of |
| 3,920 documented declarations have that separator. And spec's own docs contain |
| blank comment lines: `Add`'s is a summary, a `//`, then `z[i] = x[i] + y[i]`. |
| Hence, using a bare `//` would not be idempotent. |
| |
| **The rule.** A doc comment is spec-owned paragraphs, followed by |
| "implementation notes", followed by directive comments (`//go:fix inline` and |
| similar). Spec-owned paragraphs describe the behavior of an operation. |
| Implementation notes are details of how one target implements an operation. The |
| sections must appear strictly in this order, and the tools will validate this. |
| |
| An *implementation note* is a paragraph that starts with one of a fixed set of |
| prefixes: `Asm:`, `CPU Feature:`, `Emulated[,\n]`, `Deprecated:`, |
| `Performance:`, or `Note:`. The match is exact and case-sensitive. Only the |
| first line of a paragraph is checked, so a note can combine prefixes (`Asm: |
| VPADDD, CPU Feature: AVX`) and wrap freely. |
| |
| **Checks.** Strict ordering is necessary for idempotence, and the tools perform |
| the following checks to ensure we maintain it: (a) In an existing comment, |
| spec-owned paragraphs (those without an implementation note prefix) must appear |
| strictly before implementation notes (those with an implementation note prefix), |
| which must appear strictly before directive comments. (b) Spec's docs may not |
| contain a line starting with a note prefix, and spec stores them in canonical |
| `go/doc/comment` form. |
| |
| **Rewrite.** `specfill` rewrites a comment only when spec has an entry for the |
| declaration and check (a) passes. The new comment is the spec doc, then a blank |
| line if there are implementation notes, then those notes unchanged, then the |
| directives unchanged. In every other case the comment is left alone. This is |
| idempotent: by (b) the injected spec doc never looks like a note, so the next run |
| splits the comment at the same place and produces the same bytes. Idempotence |
| can't catch text lost on the *first* run, though, so as we transition to spec, |
| generated docs for newly covered operations must be reviewed carefully. |
| |
| **Why prefixes, not markers.** The alternative was bracketing `specfill`'s region |
| with directive comments (`//spec:doc` … `//spec:end`). Those are invisible in |
| godoc and safe by default — no marker, nothing touched — but they add lines to |
| every declaration and say nothing about *why* text is outside spec. Prefixes |
| reuse the shape the notes already have, work the same in generated and |
| hand-written files, and force each piece of non-spec text to say what kind of |
| note it is. The cost is that unmarked text defaults to spec-owned, which is why |
| every rewrite is reviewed as a diff, and why generators, once they emit no API |
| prose of their own, may not emit spec-owned text at all (see `specfill`). |
| |
| ## 1.7 Errors versus tolerated gaps |
| |
| At CL 831485, of the 782 `(receiver, name)` pairs declared for more than one |
| target, not one had divergent parameter or result *types* — but 72 disagreed on |
| parameter names and 16 on result names. So spec is *preventive, not remedial*: |
| there is no backlog of type-level fixes, and incremental migration is not racing |
| decay. |
| |
| - **Signature mismatch is an error.** For an operation spec knows about, a |
| disagreement is a bug in spec or in a generator. |
| - **A missing spec function is tolerated** until the transition completes. |
| - **Parameter and result names are checked** where the declaration names them |
| (unnamed ones are filled from spec; §1.8), for two reasons: they appear in |
| documentation and so must be consistent with the documentation's source, and |
| several past bugs have come from confused or swapped argument names. |
| - **`specfill` considers exported declarations only.** This is what makes the |
| tolerated-miss list converge: all three `ops_internal_*.go` files (amd64, |
| arm64, sve) contain zero exported declarations, so every compiler-only |
| operation falls outside `specfill`'s view without needing a category. The |
| permanent residue is small and enumerable: the `String`, `Len`, `ToArch` |
| methods and `xFromArch` functions. |
| |
| ## 1.8 Named results |
| |
| Spec names its results and its docs reference those names (`z[i] = x[i] + |
| y[i]`). Almost nothing in the shipped API names its results (SPEC-STATS.txt §2). |
| When spec's doc becomes archsimd's doc, the references to `z` in the docs won't |
| make sense unless we carry over spec's result names as well. |
| |
| Decision: **where spec covers a declaration, its parameter and result names |
| come from spec**, so they agree with the doc that also comes from spec. Naming |
| every result is *not* a goal: a name is worth having where a doc or a body uses |
| it, and naming the rest blindly serves no one. Declarations spec does not cover |
| keep whatever names they have until `spec-all` covers them. |
| |
| `specfill` enforces this without touching bodies. It rewrites unnamed |
| arguments/results in the API (including `_`) to match spec, and reports a |
| mismatch error if both sides name an argument/result and the names disagree. For |
| a bodyless declaration, all that matters is that the names agree with the doc. A |
| body can refer to named arguments but not unnamed ones, so filling unnamed ones |
| is safe, while named ones must be checked. SPEC-STATS.txt §2 counts the two |
| cases separately, the same way. |
| |
| ## 1.9 Behavioral conformance is a parallel track |
| |
| Behavioral conformance has **no hard dependency on the API documentation |
| track**. Both consume the same input — spec coverage — independently, and they |
| catch disjoint bug classes: a swapped argument in a new architecture's `Sub` has |
| an identical signature and identical docs, and only conformance sees it. |
| |
| The order for bootstrapping conformance matters. **Validate spec against amd64 |
| first** — it is the most mature target, and every failure is a spec bug unless |
| proven otherwise. There may be bugs or hardware-specific corner cases in the |
| amd64 implementation, but bugs in spec are far more likely. Once spec and amd64 |
| agree, **then flip**: spec becomes the oracle and gates arm64, wasm, SVE, and |
| any architecture added later. Getting this backwards means conformance failures |
| against correct implementations, and the obvious move is then to "fix" the |
| implementation. |
| |
| Harness requirements implied by the bug classes we actually see: per-type |
| edge-case inputs (0, ±1, MinInt, MaxInt, ±0, ±Inf, NaN, saturation boundaries) |
| crossed with random vectors — uniform random will not reliably hit saturation or |
| NaN corners; and explicit mask patterns, since an inverted mask sense passes |
| random testing about half the time. |
| |
| **This is a path we have already gone down successfully.** |
| `simd/internal/test_helpers` carries per-type input vectors — `float32s` through |
| `uint8s`, plus `bools` for masks — built by cycling a hand-picked list of |
| exactly those corners (±0, ±Inf, quiet and signalling NaN, `MaxFloat`, and the |
| 8- and 16-bit saturation edges) up to a fixed length. In |
| `archsimd/internal/simd_test/helpers_test.go`, the |
| `forSlice`/`forSlicePair`/`forSliceTriple`/`forSlicePairMasked` helpers walk |
| windows over them for the cross-product, `map1`/`map2`/`map3`/`mapCompare` build |
| the expected result elementwise — the same shape a spec body has — and |
| `grouped2` already handles the 128-bit grouping that gives `doc-explore` |
| trouble. `checkSlicesLogInput` even takes a tolerance, which is a first cut at |
| the vocabulary `impl-defined` has to settle. |
| |
| `conform-fixed` need not reuse that code literally, but it should not start over |
| either: what it adds is generating the cases from spec rather than hand-writing |
| them per operation. |
| |
| ## 1.10 Progress is measured by name, not by normalization |
| |
| `apisum` needs normalization because its job is to *test* whether names mean one |
| thing. Once we enforce that they do, **the method name is the equivalence class**, |
| exactly and for free: coverage is a set intersection, and `cmd/specstats` already |
| computes it. No name normalization, and no complex pattern matching rules to |
| maintain or throw away. |
| |
| Caveat, worth knowing rather than fixing: templated-name families like |
| `LoadInt8x16`, `LoadInt8x32`, etc, that all come from one spec function inflate |
| the denominator of any coverage metrics. That is arguably correct for measuring |
| API *surface*. Reporting both numbers — "spec has N functions covering M names" |
| — is more informative than either alone. |
| |
| --- |
| |
| # Part 2 — Task graph |
| |
| The 27 tasks fall into six **tracks**. Tracks are a way to divide the work among |
| people and to make Part 3 navigable. They are not phases and don't imply any |
| ordering on tasks. Tasks in each track generally cluster in the task dependency |
| graph, but there are inter-track dependencies. |
| |
| | | Track | What it is | Tasks | |
| |---|---|---|---| |
| | **A** | Decisions | Human judgment calls; no prerequisites, all block downstream work | `width`, `compiler-ops`, `convert-lo`, `mask-bits`, `impl-defined` | |
| | **B** | Instrumentation and enforcement | Build `specfill`, then make it authoritative | `specfill`, `fill-gen`, `named-results`, `gen-docs`, `fill-enforce` | |
| | **C** | Conformance | Behavioral testing against `simdref` | `ci-audit`, `conform-fixed`, `validate-amd64`, `conform-scalable`, `conform-flip` | |
| | **D** | Spec content | Writing spec functions, docs, and bodies — the shared bottleneck | `doc-explore`, `spec-common`, `spec-all` | |
| | **E** | Retiring the old inputs | The five non-spec doc sources, then `categories.yaml` | `comments-yaml`, `doc-triage`, `handwritten-fill`, `template-docs`, `cat-rehome`, `compiler-ops-impl`, `cat-delete` | |
| | **F** | simdgen internals | The signature refactor, then sourcing signatures from spec | `sig-refactor`, `sig-from-spec` | |
| |
| ## Dependencies |
| |
| A task is startable when every prerequisite is ticked. Hard dependencies are |
| listed as `Needs`; soft dependencies — work is possible but wasteful or |
| lower-value without them — as `Prefers`. |
| |
| Tasks are named, not numbered, so that a task's identity survives inserting |
| another one next to it, and so that a CL saying "implements `cat-rehome`" still |
| means something a year later. Ordering lives in the row order instead: the table |
| is topologically sorted (both `Needs` and `Prefers` count as edges), with ties |
| broken by track, so reading it top to bottom is a legal order to do the work in, |
| and `cmd/spectasks` fails if an edit breaks that. Add a task by putting its row |
| wherever its prerequisites and track allow. |
| |
| | ✓ | T | Task | What it is | Needs | Prefers | |
| |---|---|---|---|---|---| |
| | [x] | A | `width` | Resolve scalable width | — | — | |
| | [ ] | A | `compiler-ops` | Compiler-only ops decision | — | — | |
| | [ ] | A | `convert-lo` | `ExtendLo*`/`ConvertLo*` shape and naming | — | `width` | |
| | [ ] | A | `mask-bits` | Mask bitmap load/store shape and naming | — | `width` | |
| | [ ] | A | `impl-defined` | Implementation-defined behavior policy | — | — | |
| | [x] | B | `specfill` | API walker + report mode | — | — | |
| | [x] | B | `fill-gen` | Wire `specfill` into the generators | `specfill` | — | |
| | [x] | B | `named-results` | Reconcile parameter and result names with spec | `fill-gen` | — | |
| | [x] | B | `gen-docs` | Inject spec docs into generated output | `fill-gen` | `named-results` | |
| | [x] | B | `fill-enforce` | Signature + name mismatch becomes an error | `named-results` | — | |
| | [x] | C | `ci-audit` | CI/hardware audit | — | — | |
| | [ ] | C | `conform-fixed` | Conformance harness, fixed-width types | `ci-audit` | `specfill` | |
| | [ ] | C | `conform-scalable` | Conformance harness, scalable types | `width`, `conform-fixed` | — | |
| | [x] | D | `doc-explore` | Doc-mechanism exploration on a hard family | — | — | |
| | [x] | D | `spec-common` | Spec the common operations | `width`, `doc-explore` | `specfill` | |
| | [ ] | C | `validate-amd64` | Validate spec bodies against amd64 | `conform-fixed`, `spec-common`, `impl-defined` | — | |
| | [ ] | C | `conform-flip` | Flip conformance direction | `validate-amd64`, `conform-scalable` | — | |
| | [ ] | D | `spec-all` | Bulk spec migration | `spec-common`, `convert-lo`, `mask-bits` | `specfill`, `fill-enforce` | |
| | [ ] | E | `doc-triage` | Hand-written doc conflict triage | `specfill` | `spec-common` | |
| | [x] | E | `comments-yaml` | Drop spec-covered docs from `midway/comments.yaml` | `gen-docs`, `spec-common` | `fill-enforce` | |
| | [ ] | E | `handwritten-fill` | Enable `-w` on hand-written files | `gen-docs`, `doc-triage` | — | |
| | [ ] | E | `template-docs` | Retire generator-supplied API docs | `gen-docs`, `spec-all` | — | |
| | [ ] | E | `cat-rehome` | Re-home `categories.yaml` non-doc fields | `template-docs` | — | |
| | [ ] | E | `compiler-ops-impl` | Implement compiler-only ops representation | `compiler-ops`, `cat-rehome` | — | |
| | [ ] | F | `sig-refactor` | simdgen signature refactor | — | — | |
| | [ ] | F | `sig-from-spec` | simdgen sources signatures from spec | `sig-refactor`, `cat-rehome`, `compiler-ops-impl` | — | |
| | [ ] | E | `cat-delete` | Delete `categories.yaml` | `sig-from-spec` | — | |
| |
| **Critical path.** [SPEC-TASKS.dot](SPEC-TASKS.dot) draws it in red: every task |
| on some longest chain of `Needs` edges. Slipping any of them slips the whole |
| project. `cmd/spectasks` derives it from the table, so it is not restated here. |
| |
| Track B has no prerequisites, so it can start immediately and run to completion |
| at its own pace — though `gen-docs` gates three of track E's tasks. |
| `sig-refactor` is independent of everything until `sig-from-spec`. |
| |
| ## Milestones |
| |
| Milestones carry no work of their own; they are the points worth announcing. A |
| milestone is reached when its tasks are ticked, so there is no separate state to |
| maintain and nothing here to keep in sync. |
| |
| | Milestone | Reached after | What it means | |
| |---|---|---| |
| | **M1 — Enforced** | `fill-enforce` | Spec is authoritative for what it covers: a name or signature that drifts from it fails the build, in both packages, on all four targets. | |
| | **M2 — Documented** | `comments-yaml`, `handwritten-fill`, `spec-all`, `template-docs` | Every operation's doc comment in both public packages derives from spec — SPEC-STATS.txt's "docs sourced from spec" reads 100%. | |
| | **M3 — Gated** | `conform-flip` | Conformance runs spec-first, so an architecture added later must match spec before it can land. | |
| | **M4 — Complete** | `cat-delete` | `categories.yaml` is gone and spec is the only hand-written description of the API; see the section below. | |
| |
| ## The graph |
| |
| [SPEC-TASKS.dot](SPEC-TASKS.dot) draws the two tables above; the rendered graph |
| carries its own key. Track is node colour rather than a cluster, because a |
| cluster has to be laid out contiguously and that fights the left-to-right |
| dependency order the graph exists to show. |
| |
| dot -Tsvg SPEC-TASKS.dot -o SPEC-TASKS.svg |
| |
| It is generated, and its generator is also a checker. This document states its |
| dependencies three times — Part 2's table, Part 3's `Needs:`/`Prefers:` lines, |
| and Part 3's `Blocks:` lines, which are the same edges reversed — so |
| `go test ./cmd/spectasks` fails if they disagree, or if the graph has not been |
| regenerated since. It also derives the critical path. |
| |
| ## Start here |
| |
| The startable tasks are the ones with no unticked `Needs` in the dependency |
| table. For a team of one to three, this is a reasonable opening allocation, |
| covering every task with no `Needs` at all: |
| |
| - **Strongest on spec's type model:** `width`, then `convert-lo` and |
| `mask-bits`. These root the longest chain and cost the most if deferred. |
| - **Owns API semantics:** `impl-defined` and `compiler-ops` — pure |
| decisions, no code, and everything downstream waits on them. |
| - **Likes tooling:** `specfill`, then `fill-gen` and `named-results`. |
| - **Working alone on something large:** `sig-refactor`. No prerequisites, |
| verifiable byte-for-byte at every step. |
| - **Cheap and worth doing early regardless:** `doc-explore` and `ci-audit`. |
| |
| ## What "done" looks like |
| |
| After `cat-delete`, `simd/internal/spec` is the only hand-written description of |
| the API: every operation's name, signature, and documentation derives from it, |
| in both public packages and on all four targets, and `specfill` fails the build |
| if any of them drift. `categories.yaml` is gone. `midway/comments.yaml` |
| documents only what spec does not describe: the vector types, `VectorSize`, |
| `String` and `Len`. wasmgen and tmplgen no longer carry doc strings. |
| Hand-written implementations remain hand-written, but their operation doc |
| comments are generated. |
| |
| simdgen still exists and is still unification-based — §1.1 explains why. What |
| changes is its job: the unifier no longer defines the API, it only expands one |
| `go_GOARCH.yaml` entry into all the concrete machine instructions matching it. |
| `go_GOARCH.yaml` stays, and so do its quirk fields, because instruction |
| irregularity is real and has to live somewhere. The compiler backend generation |
| is untouched throughout. |
| |
| --- |
| |
| # Part 3 — Task details |
| |
| Many completion criteria are "generated output is byte-identical", checked with |
| `go run . -diff` from `_gen`. That runs every generator, including simdgen for |
| all three of its targets, so it needs the XED and ARM ISA data first — see |
| `fetch-xed.sh` and `fetch-arm64.sh`. There is no `-arch` flag on the driver; |
| `-arch` belongs to `simdgen` when run directly. |
| |
| ## Track A — Decisions |
| |
| Human judgment calls. None has prerequisites, and each blocks downstream work — |
| which is why they lead. Each is small in code and large in consequence. |
| |
| ### `width` — Resolve scalable width |
| **Done:** [x] · **Needs:** — · **Blocks:** `spec-common`, `conform-scalable` · **Risk:** `risk-concrete-width` |
| |
| `simd/internal/spec/types.go` hardcodes `scalableWidth = 4096`, and `simdref` is |
| built at that width, while `archsimd`'s SVE uses the runtime `vl()`. A |
| conformance test of scalable types therefore compares vectors of different |
| lengths. Spec's own TODO flags this and it is unresolved. |
| |
| 4096 bits in particular causes trouble for operations that index into vectors, |
| such as shuffles, which is why every platform other than RISC-V RVV is 2048 bits |
| or less. Resolving this task may require deciding whether we'll ever support |
| vectors more than 2048 bits wide. |
| |
| Note what is *not* wrong: spec's type system was designed for scalable widths and |
| handles them. The gap appears only when spec is used as a concrete conformance |
| oracle, and the open question is narrow — whether spec can execute at a |
| caller-chosen width without disturbing the generic type machinery. |
| |
| This is first because the fix may change spec's type model — making width a |
| runtime parameter rather than a compile-time constant — which reaches every spec |
| function body. Resolving it after a hundred bodies exist is far worse than |
| resolving it now, while there are few (SPEC-STATS.txt §1). |
| |
| **Commits.** (1) Write up the options with their consequences for `Vec`, |
| `makeVec`, `lanes`, `refgen`, and the conformance harness; seen so far: |
| parameterize width at spec execution, make `simdref` width-configurable at |
| construction, or scope conformance to fixed widths initially and defer. (2) |
| Implement the chosen approach in `internal/spec`. (3) Regenerate |
| `internal/simdref` and confirm `cmd/specls` output is unchanged, or explain what |
| changed. |
| |
| **Done when** a scalable-typed operation can be executed through `simdref` at a |
| width matching a chosen target, and the choice is documented in |
| `internal/spec/doc.go`. |
| |
| ### `compiler-ops` — Compiler-only ops decision |
| **Done:** [ ] · **Needs:** — · **Blocks:** `compiler-ops-impl` |
| |
| Decide how the compiler-only entries — those that produce no exported |
| declaration (SPEC-STATS.txt §4) — and the `noTypes` / `noGenericOps` switches |
| are represented once `categories.yaml` is gone. This is the only hard blocker in |
| the chain without an obvious answer, so make it early. |
| |
| Note §1.7: because `specfill` is scoped to exported declarations, this does |
| *not* block the checker or conformance tracks. |
| |
| The design note must also say where the following go once `categories.yaml` is |
| gone: the compiler-only entries' `documentation:`, which feeds the doc comments |
| of unexported methods in `archsimd/ops_internal_*.go`; `commutative`, `signed` |
| and `out` on those entries; and the `go_arm64.yaml` use of `noGenericOps`. This |
| documentation is strictly internal-facing, so it is held to a lower standard |
| than user-facing API docs. It need not come from spec, go through `specfill`, or |
| satisfy M2. Wherever it lands is acceptable as long as it is kept next to the |
| compiler-only op's other description. |
| |
| **Commits.** One design note in this document or in `internal/spec/doc.go`, |
| listing all compiler-only operations from `_gen/simdgen` (`grep -h "go: [a-z]" |
| ops/*/categories.yaml | sort -u`; also the capitalized `noTypes` entries) and |
| stating for each family where its description, documentation, and remaining |
| fields live once `categories.yaml` is gone. Candidates: a `//specgen:unexported` |
| directive; a fourth simdgen input holding compiler-only ops; or leaving them in |
| `go_GOARCH.yaml` on the grounds that they are per-architecture lowering details |
| rather than API. |
| |
| **Done when** the decision is written down, including documentation and all |
| remaining fields, and `compiler-ops-impl` could be implemented from it without |
| further design work. |
| |
| ### `convert-lo` — `ExtendLo*` / `ConvertLo*` shape and naming |
| **Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects` |
| |
| Settle how the low-elements widening family is shaped and named, across both |
| fixed-width and scalable vectors. An API design decision, not a mechanical fix. |
| Three entangled problems: |
| |
| 1. **Shape.** `ExtendLoLToZ` declares `x Vec[E,W]` and `z Vec[zE,W]` — the same |
| total width. The shipped API widens instead: `Int8x16.ExtendLo8ToInt64()` |
| takes 128 bits and returns 512. Spec therefore misses 8 real declarations and |
| invents width-preserving ones no platform has. |
| 2. **Naming.** `//specgen:name ExtendLo{zL}To{zE}` embeds the *result lane |
| count*, so the same conceptual operation gets a different name at each width. |
| The API has settled on 128-bit receivers only, which hides this. |
| 3. **Scalable.** The `converts.go` TODO already notes the literal `zL` cannot |
| work for scalable vectors, and suggests dropping the number entirely since it |
| is implied by the receiver and target types — e.g. `Uint8x16.ExtendLo2ToUint64` |
| is really the low *eighth*, not the low half, so the number is arguably |
| misleading even where it works. |
| |
| Any renaming here is API-visible and should be decided before `spec-all` |
| migrates the Converts family, which is where the bulk of `categories.yaml`'s |
| Converts documentation lands. |
| |
| **Commits.** (1) Write up the options and their consequences for fixed-width and |
| scalable vectors, including whether the lane count stays in the name. (2) |
| Re-shape `ExtendLoLToZ`, and re-check `ConvertLoLToZ`, whose `zL<xL` constraint |
| currently admits conversions no hardware offers. (3) If the API name changes, |
| land the rename through the generators as its own CL, separate from the spec |
| change. |
| |
| **Done when** `cmd/specls` produces exactly the `ExtendLo*` declarations the API |
| has, plus whatever maximalist extras are deliberate, and the scalable naming |
| question is answered in `internal/spec/doc.go`. |
| |
| ### `mask-bits` — Mask bitmap load/store shape and naming |
| **Done:** [ ] · **Needs:** — · **Prefers:** `width` · **Blocks:** `spec-all` · **Risk:** `risk-spec-defects` |
| |
| Settle the scalable counterpart of fixed-width `ToBits` and `MaskNxMFromBits` |
| (e.g. `Mask8x16FromBits`), which today are `Mask*s.Store` and `LoadMask*s`. Also |
| settle what `Store` and `ToBits` mean on portable `simd.Mask*s`, whose midway |
| docs currently describe element semantics. |
| |
| The API declares `Store(bits []uint16)` only on SVE's scalable masks |
| (`archsimd/types_sve.go`), paired with `LoadMask8s(bits []uint16)` etc. It |
| stores SVE predicate bits at byte granularity. No fixed-width mask has `Store`; |
| fixed-width masks have `ToBits() UintN` instead (spec: `MaskToBits`, |
| `internal/spec/masks.go`, constrained to `FixedWidth`). So `Mask*s.Store` has a |
| different meaning from every other `Store` — a candidate violation of the "one |
| name, one meaning" rule. Its bit layout is SVE's representation, not a per-lane |
| one, which arguably fails §1.3's "claim about the operation's own semantics" |
| test. |
| |
| **Commits.** (1) Write up the options and their consequences for fixed-width and |
| scalable masks, and for the portable package. (2) Change spec to match. (3) If |
| the API name changes, land the rename through the generators as its own CL. |
| |
| **Done when** SPEC-STATS.txt §5 lists no mask `Store` defect, and the choice is |
| recorded in `internal/spec/doc.go`. |
| |
| ### `impl-defined` — Implementation-defined behavior policy |
| **Done:** [ ] · **Needs:** — · **Blocks:** `validate-amd64` · **Risk:** `risk-impl-defined-behavior` |
| |
| Decide what counts as implementation-defined behavior in the SIMD API, and how |
| spec expresses it so conformance testing stays meaningful. Architectures |
| legitimately differ on corners — the last bits of a float result, out-of-range |
| shuffle indices, saturation edges, NaN propagation and signed zero. Spec's |
| `math.go` already carries a TODO on `DotProductPairs` overflow that names the |
| problem and floats an "architecture dependent" designated panic as one possible |
| answer. |
| |
| Two halves, and both need answering: |
| |
| 1. **What is implementation-defined?** A per-operation API-semantics judgment. |
| Costly in both directions: declare too much and conformance stops catching |
| real bugs; declare too little and correct implementations fail against spec. |
| 2. **How is it written down?** There are at least two major categories of |
| implementation-defined behavior: exceptional-value cases (e.g., overflow |
| behavior is fully undefined) and precision (all architectures roughly agree, |
| but maybe not to the last digit). Spec bodies return exact values today, and |
| there is no vocabulary for "any value here", "within N ulp", or "unspecified |
| but deterministic per platform". Whatever is chosen must be understood by |
| both `refgen` and the conformance harness, and it changes what a spec body |
| looks like — which is why this wants settling before the bulk of them are |
| written. The existing suite's `checkSlicesLogInput` already takes a per-call |
| tolerance (§1.9), which is prior art for the numeric half; it says nothing |
| about the "unspecified but deterministic per platform" half. |
| |
| **Commits.** (1) Survey the known cases — `DotProductPairs` overflow, float |
| conversion rounding, shuffle index out of range, saturation edges, NaN/±0 |
| handling — classifying each as specified, implementation-defined, or undefined. |
| (2) Choose and document the spec vocabulary in `internal/spec/doc.go`. (3) |
| Implement it far enough that one operation with a genuinely |
| implementation-defined corner round-trips through `refgen` and the harness. |
| |
| **Done when** an operation with an architecture-dependent corner can be written |
| in spec, generated into `simdref`, and conformance-tested without a false failure |
| on any supported target. |
| |
| ## Track B — Instrumentation and enforcement |
| |
| Build `specfill`, then make it authoritative. Independent of tracks C, D and F; |
| `gen-docs` gates three of track E. |
| |
| ### `specfill` — API walker + report mode |
| **Done:** [x] · **Needs:** — · **Blocks:** `fill-gen`, `doc-triage` |
| |
| The instrument the rest of the plan reads from. |
| |
| `specfill` builds the whole of `specdoc.Fill` and `cmd/specfill` — the walk, the |
| spec index, the comparison, and the rewrite of doc comments and parameter/result |
| names. `fill-gen` wires it into generators with rewriting off; `named-results` |
| and `gen-docs` then turn on name filling and doc injection there, and |
| `handwritten-fill` turns `-w` on for hand-written files. |
| |
| **Interfaces.** `specdoc.Fill` is the library entry point — Go source in, Go |
| source out: `func Fill(src []byte, idx *specgen.Index, opts Options) ([]byte, error)`. |
| Generators call it on their buffer before writing (§1.5). When verification |
| checks fail, `Fill` returns a `*Report` as an error containing the findings |
| enumerated below. Callers can recognize this error via `errors.As` if they |
| choose to inspect or combine findings across files. Option fields: |
| |
| - **RejectUnknown** controls whether an exported declaration with no spec entry |
| is an error (recorded in `Report.UnknownDecls`). Off during the transition; |
| generators flip it on after `spec-all`, so that adding an operation without |
| writing its spec function fails at generation time rather than silently escaping |
| coverage. |
| - **AllowDocRewrite** controls whether a declaration spec covers may already |
| carry spec-owned paragraphs (§1.6). When false (default), each one that does |
| is an error (recorded in `Report.UnexpectedDocs`); implementation notes and |
| directives are always allowed. Once generators emit no API prose of their own |
| (`template-docs`), this is their guard against silent doc loss |
| (`risk-silent-doc-loss`): a generator paragraph missing its note prefix is an |
| error rather than something `Fill` quietly replaces. Until then generators set |
| it to true. Hand-written files carry prose by design, so `cmd/specfill` always |
| sets it to true and relies on `-diff` and its `-w` guards instead. |
| - **AllowNameMismatches** controls whether name mismatches are tolerated. When |
| false (default), they are errors. Generators can set it to true during the |
| transition, until `named-results` reconciles them. |
| |
| Two more fields select what `Fill` rewrites, for callers that want only part of |
| it: **NoFillDoc** leaves doc comments alone, and **NoFillNames** leaves unnamed |
| arguments unnamed (and so reports them as name mismatches). By default `Fill` |
| does both. Finally, the **Filename** field labels positions in findings. |
| |
| **Names.** `Fill` fills and checks names as §1.8 describes: an argument the |
| declaration leaves unnamed (or names `_`) takes spec's name, and a named |
| argument whose name differs from spec's is a mismatch. It rewrites a |
| declaration's names only when that declaration has no type or name mismatch, so |
| a mismatch has to be fixed before the rest of its declaration's names are |
| filled. |
| |
| `cmd/specfill` is a `gentools`-based command that applies `specdoc` to |
| hand-written files: it walks the non-test `.go` files in `simd` and `archsimd`, |
| skips generated ones (their generators call `Fill` themselves), calls `Fill` on |
| each with `AllowDocRewrite`, and prints the combined report. `-diff` shows what |
| a rewrite would change. With `-w`, rewrite in place; `-w` aborts without |
| rewriting when the report has any findings, or if there are uncommitted changes, |
| so its git diff is the complete story of what it did. The `-no-fill-doc` and |
| `-no-fill-names` flags set the corresponding options. CI enforcement is a `go |
| test` in this package — `fill-enforce` wires it — not a command invocation. |
| |
| **Report.** The report is the permanent enforcement layer — every |
| item in it remains useful after the transition as the API continues to evolve, |
| and is what `fill-enforce` runs in CI. Coverage and name relations are |
| transition metrics that converge to their targets; they stay in |
| `cmd/specstats` / SPEC-STATS.txt and are not part of this report. |
| |
| The report holds only errors; it does not preview rewrites. Every consumer of |
| `specdoc` is a `gentools` program, `cmd/specfill` included, so `-diff` already |
| shows exactly what a rewrite would change against the files on disk, including |
| any text it would drop. That diff is how rewrites are reviewed, and a report that |
| recomputed it would only be a second copy to keep in sync. |
| |
| The possible errors are: |
| |
| 1. **Type signature mismatches** — parameter or result types, or their |
| counts, disagree between spec and the declaration the walker found. |
| 2. **Name signature mismatches** — a named parameter or result whose name |
| differs from spec's. Unnamed ones are filled rather than reported, unless |
| `NoFillNames` is set. Omitted when `AllowNameMismatches` is set. |
| 3. **Check (a) violations** — §1.6 ordering violations in declarations spec |
| covers. `Fill` leaves such a comment as it is. Declarations spec does not |
| cover are not checked: the ordering rule exists so rewrites are idempotent, |
| and `Fill` never rewrites them. Hand-written files in particular may hold |
| declarations spec will never cover. Each declaration is checked from the |
| moment spec covers it. |
| 4. **Unknown declarations** — exported declarations with no spec entry, when |
| `RejectUnknown` is set. |
| 5. **Unexpected doc comments** — declarations spec covers that carry |
| spec-owned paragraphs, when `AllowDocRewrite` is false. |
| |
| **Commits.** (1) `specgen`: add an index over `Load()`'s result keyed by `(recv, |
| name)`; add a function that classifies comment paragraphs as spec-owned or |
| implementation note per §1.6; reject spec docs that break check (b). (2) New |
| package `_gen/specdoc` and new command `_gen/cmd/specfill`: AST declaration |
| walker, signature and comment ordering comparison, report findings implementing |
| the `error` interface, and `AllowDocRewrite` / `AllowNameMismatches` options. |
| Signature comparison translates the parsed `*ast.FuncDecl` into spec's own |
| representation — its types into `specexpr` types, which suffices because no |
| qualified types exist (§1.4) — and compares that structurally against the |
| `specgen.Func`. (3) Rewrite engine in `specdoc.Fill` (both doc comments and |
| signature names) and in-place rewriting (`-w`, `-no-fill-doc`, `-no-fill-names`) |
| in `cmd/specfill`. |
| |
| **Done when** `specfill`, run on the current tree, writes nothing (without `-w`) |
| and prints all report findings in position order. |
| |
| `SPEC-STATS.txt` is the sanity check for signatures: `cmd/specstats` computes |
| the same type and name mismatches, counting names the way `Fill` does (§1.8), so |
| `specfill`'s report should agree with it line for line on those. Treat any |
| disagreement as something to explain before shipping. Coverage and name |
| relations stay in `specstats` as transition metrics; once they reach their |
| targets, `specstats` can be retired. |
| |
| ### `fill-gen` — Wire `specfill` into the generators |
| **Done:** [x] · **Needs:** `specfill` · **Blocks:** `named-results`, `gen-docs` |
| |
| Each generator passes its buffer through `specfill` before writing, with all |
| rewriting off for now: `NoFillNames` (`named-results` turns name filling on) and |
| `NoFillDoc` (`gen-docs` turns doc injection on). Generators also set |
| `AllowNameMismatches`, because names still disagree with spec until |
| `named-results`, and `AllowDocRewrite`, because they still emit API prose of |
| their own until `template-docs`. |
| |
| So this task only wires the generators up, and generated output does not |
| change. What does change is that `Fill`'s remaining checks now run at |
| generation time: a type mismatch or a check (a) violation in generated output |
| fails generation. Neither can be tolerated by an option, and as of `specfill`'s |
| completion there are none in generated output. |
| |
| **Commits.** (1) Add a narrow post-processing hook to `gentools` that does not |
| make gentools depend on specgen (see §1.5). (2) Wire simdgen. (3) Wire tmplgen, |
| wasmgen, midway. |
| |
| **Done when** all four generators route through `specfill`, and generated |
| output is byte-identical to the previous tree. |
| |
| ### `named-results` — Reconcile parameter and result names with spec |
| **Done:** [x] · **Needs:** `fill-gen` · **Blocks:** `fill-enforce` |
| |
| Every declaration spec covers carries spec's parameter and result names. See |
| §1.8. Declarations spec does not cover are out of scope; they gain names when |
| `spec-all` covers them. |
| |
| SPEC-STATS.txt §2 has the counts, split the way `specfill` treats them: |
| |
| - **Named arguments whose names disagree with spec**, in generated or |
| hand-written code. Each is fixed in whichever of spec or the API has the worse |
| name — spec, a generator's template or input, or a hand-written file; either |
| way the diff touches only names, and doc text that mentions them. These come |
| first: `Fill` fills no names in a declaration with a mismatch. |
| - **Unnamed arguments**, which `Fill` fills from spec. In generated code, that |
| means turning name filling on in the generators. In hand-written files, |
| `specfill -w -no-fill-doc` fills them; `-w` refuses while the report has |
| findings, so this also needs any check (a) violations in hand-written files |
| cleared (they are also `doc-triage`'s). |
| |
| Cross-target parameter-name divergence (also SPEC-STATS.txt §2) is only fixed |
| here for pairs spec covers. The rest reach zero as `spec-all` covers them, when |
| each new spec function has to agree with every target at once. |
| |
| **Commits.** (1) Reconcile name mismatches, grouped by generator or operation |
| family, each commit with its regenerated output. (2) Stop setting `NoFillNames` |
| and `AllowNameMismatches` in the generators, with the regenerated output: name |
| filling is on, and a name mismatch in generated output fails generation. (3) |
| Fill unnamed arguments in hand-written files with `specfill -w -no-fill-doc`. |
| |
| **Done when** SPEC-STATS.txt §2's name mismatches and unnamed counts are zero, |
| and the generators run without `NoFillNames` or `AllowNameMismatches`. |
| |
| ### `gen-docs` — Inject spec docs into generated output |
| **Done:** [x] · **Needs:** `fill-gen` · **Prefers:** `named-results` · **Blocks:** `comments-yaml`, `handwritten-fill`, `template-docs` · **Risk:** `risk-silent-doc-loss` |
| |
| Turn on doc injection in the generators by dropping `NoFillDoc`. Generators keep |
| setting `AllowDocRewrite` until `template-docs` removes their own prose. |
| |
| Prefer landing this after `named-results`: spec's docs refer to argument names |
| (§1.8), so injecting them before the generators fill those names leaves docs |
| that mention names the signature does not bind. |
| |
| Expect a real diff, not a no-op: spec already overlaps several hundred |
| `(receiver, name)` pairs (SPEC-STATS.txt §1), and almost none of the existing |
| doc text matches spec's (SPEC-STATS.txt §3). So this rewrites that many doc |
| comments. Use `go run . -diff` first to size the change exactly. |
| |
| Before any generator's docs are injected, its implementation notes must follow |
| §1.6, or the first rewrite silently drops them. Known gaps: some notes say `CPU |
| Features:` or omit a colon; simdgen's performance caveats ("A non-constant value |
| of shift may result in significantly worse performance", and the `SetElem` |
| jump-table line) have no prefix, and the `SetElem` one sits directly above its |
| `Asm:` line with no blank line between them. `specfill`'s check (a) violations |
| and a `-diff` of the output with injection on are the list to work from. |
| |
| **Commits.** (1) Fix note prefixes in simdgen, tmplgen, wasmgen and midway |
| templates — `Performance:` for the caveats, a blank line before every note — |
| with the regenerated output; the diff should touch only note lines. (2) Turn on |
| doc injection in simdgen. (3) Turn it on in tmplgen, wasmgen, midway. Commits |
| (2) and (3) include their regenerated output; verify that output changes only |
| for operations spec covers, and review every changed doc comment to make sure |
| nothing was unintentionally dropped. (4) If `fill-enforce` is already done, drop |
| `NoFillDoc: true` from `cmd/specfill.TestFill` for generated files to start |
| enforcing documentation matches. |
| |
| **Done when** generated output has no check (a) violations, all four generators |
| inject docs, nothing was dropped without review, and the only differences |
| against the previous tree are doc comments for operations spec covers. |
| |
| ### `fill-enforce` — Signature + name mismatch becomes an error |
| **Done:** [x] · **Needs:** `named-results` |
| |
| For operations spec covers, disagreement fails. Missing spec functions remain |
| tolerated. Note this does **not** depend on `sig-refactor`: `specfill` compares |
| emitted source, not simdgen's internal model. |
| |
| Generated output is already checked as it is generated: type mismatches fail |
| generation from `fill-gen` on, and name mismatches from `named-results` on. |
| That misses hand-written files, and generated files edited by hand without |
| rerunning their generator. This task closes both with a `go test` in |
| `cmd/specfill` that runs `Fill` over every non-test `.go` file in `simd` and |
| `archsimd`, generated or not, and fails on any finding. It sets |
| `AllowDocRewrite`, since those files carry spec-owned docs by design, whether |
| injected or hand-written. |
| |
| **Commits.** One, adding the test. |
| |
| **Done when** a deliberately introduced signature or argument-name change in any |
| generated or hand-written exported declaration fails the build. |
| |
| ## Track C — Conformance |
| |
| Behavioral testing against `simdref`. Parallel to everything else; shares only |
| the spec-coverage bottleneck. |
| |
| ### `ci-audit` — CI/hardware audit |
| **Done:** [x] · **Needs:** — · **Blocks:** `conform-fixed` · **Risk:** `risk-conformance-env` |
| |
| Determine what can actually execute tests for amd64, arm64, wasm, and SVE — |
| natively or emulated. The conformance suite's value for other architectures |
| depends on it. |
| |
| "Execute tests" means: build `simd` and `archsimd` for the target and run |
| `go test` against real or emulated hardware, at a speed that tolerates a few |
| thousand generated conformance cases. SVE needs SVE-capable hardware or QEMU, and |
| wasm needs a runtime. |
| |
| **Done when** one document or issue records, per target, whether it can run |
| today, on what, how fast, and what it would take if not — marking each runnable, |
| runnable-with-work (with the work named), or not runnable, so `conform-fixed` |
| knows which targets it is writing tests for. |
| |
| **Results** |
| |
| | amd64 | gotip-linux-amd64 | gotip-linux-amd64_avx512 | |
| | ---------------- | ----------------- | ------------------------ | |
| | AVX | ✔ | ✔ | |
| | AVX2 | ✔ | ✔ | |
| | AVX512 | ✘ | ✔ | |
| | AVX512BITALG | ✘ | ✔ | |
| | AVX512GFNI | ✘ | ✔ | |
| | AVX512VAES | ✘ | ✔ | |
| | AVX512VBMI | ✘ | ✔ | |
| | AVX512VBMI2 | ✘ | ✔ | |
| | AVX512VNNI | ✘ | ✔ | |
| | AVX512VPCLMULQDQ | ✘ | ✔ | |
| | AVX512VPOPCNTDQ | ✘ | ✔ | |
| | AVXAES | ✔ | ✔ | |
| | AVXPCLMULQDQ | ✔ | ✔ | |
| | AVXVNNI | ✘ | ✔ | |
| | FMA | ✔ | ✔ | |
| | SHA | ✘ | ✔ | |
| | VAES | ✘ | ✔ | |
| | VPCLMULQDQ | ✘ | ✔ | |
| |
| For amd64, use the gotip-linux-amd64_avx512 gomote for testing. |
| |
| | arm64 | gotip-linux-arm64 | qemu-aarch64 -cpu neoverse-n2 | qemu-aarch64 -cpu neoverse-v1 | |
| | --------------- | ----------------- | ----------------------------- | ----------------------------- | |
| | PMULL | ✔ | ✔ | ✔ | |
| | SVE | ✘ | ✔ | ✔ | |
| | SVE2 | ✘ | ✔ | ✔ | |
| | SVE vector size | n/a | 128 | 256 | |
| |
| `qemu-aarch64 -cpu neoverse-n1` (which matches the capabilities of |
| gotip-linux-arm64) is roughly 7x slower than `gotip-linux-arm64`, so well within |
| the range of practical conformance testing. |
| |
| For NEON, use the gotip-linux-arm64 gomote for testing. For SVE, use |
| `qemu-aarch64 -cpu neoverse-v1`. |
| |
| | wasm | js-node | wasip1-wazero | |
| | ------- | ------- | ------------- | |
| | SIMD128 | ✔ | ✔ | |
| |
| `node` is about 3x faster than `wazero`. |
| |
| For wasm, use `node` for testing. |
| |
| ### `conform-fixed` — Conformance harness, fixed-width types |
| **Done:** [ ] · **Needs:** `ci-audit` · **Blocks:** `validate-amd64`, `conform-scalable` · **Prefers:** `specfill` |
| |
| Generated tests comparing `archsimd` against `internal/simdref`. The generator's |
| input is spec (hand-written source), so this is not a generator reading another |
| generator's output. |
| |
| **Commits.** (1) Input generation: start from |
| `archsimd/internal/simd_test/helpers_test.go`, which already has the per-type |
| edge-case vectors, the mask patterns and the window-walking helpers (§1.9); |
| decide whether to lift them into a package the generated tests can import. (2) |
| Test generator producing one test per `(recv, name)` spec covers. (3) Wire into |
| the SIMD test suite, skipping operations whose spec body panics. |
| |
| **Done when** every operation spec covers, on every target `ci-audit` marked |
| runnable, is exercised against `simdref` with both edge-case and random inputs, |
| and failures name the operation, the input, and both results. |
| |
| ### `conform-scalable` — Conformance harness, scalable types |
| **Done:** [ ] · **Needs:** `width`, `conform-fixed` · **Blocks:** `conform-flip` |
| |
| Extend `conform-fixed`'s harness to the scalable types (`Int8s` … `Float64s`, |
| `Mask8s` … `Mask64s`) on SVE. Separate from `conform-fixed` because it cannot be |
| written until `width` settles how spec executes at a concrete width: the harness |
| must construct `simdref` values at the width the hardware reports through `vl()` |
| rather than at spec's fixed `scalableWidth`. |
| |
| **Commits.** (1) Teach input generation to size vectors from the runtime vector |
| length. (2) Extend the test generator to scalable receivers. (3) Wire into the |
| suite, gated on SVE availability per `ci-audit`. |
| |
| **Done when** scalable operations spec covers are conformance-tested on SVE at |
| the hardware's actual vector length. |
| |
| ### `validate-amd64` — Validate spec bodies against amd64 |
| **Done:** [ ] · **Needs:** `conform-fixed`, `spec-common`, `impl-defined` · **Blocks:** `conform-flip` |
| |
| Run conformance with amd64 as the oracle. Every failure is a spec bug until |
| proven otherwise. See §1.9. |
| |
| **Commits.** One per batch of spec fixes; keep spec changes separate from harness |
| changes so the reason for each is legible. |
| |
| **Done when** the harness passes on amd64 for every operation spec covers, with |
| every remaining difference either fixed in spec or recorded as |
| implementation-defined under `impl-defined`'s policy. |
| |
| ### `conform-flip` — Flip conformance direction |
| **Done:** [ ] · **Needs:** `validate-amd64`, `conform-scalable` |
| |
| Spec becomes the oracle for every target, not just amd64. The second half is the |
| contributor-facing note — likely in `internal/spec/doc.go` and the SIMD README — |
| describing the workflow: add the operation to spec if it is not there, implement |
| it, run conformance. |
| |
| **Done when** conformance runs and fails the build on all targets `ci-audit` |
| marked runnable, and the workflow is written down where someone adding an |
| architecture will find it. |
| |
| ## Track D — Spec content |
| |
| Writing spec functions, docs, and bodies. The shared bottleneck the other tracks |
| draw on. |
| |
| ### `doc-explore` — Doc-mechanism exploration on a hard family |
| **Done:** [x] · **Needs:** — · **Blocks:** `spec-common` · **Risk:** `risk-doc-template-ceiling` |
| |
| As of CL 831485, roughly one doc line in eight in `categories.yaml` carries |
| shape-specific literals — `[x0, x1, x2, x3, ...]`, `128-bit`, `0b_11_01_00_10` — |
| concentrated in `AddSub`'s `Concat*Pairs` and `Moves`' |
| `concatSelectedConstant*`. Those docs cannot be produced by substituting `{vL}` |
| into one template; each shape needs its own text. |
| |
| The expected mechanism is multiple spec functions sharing one `//specgen:name`, |
| which specgen supports because `funcKey` is `(recv, name)` and the constraint |
| sets are disjoint. **Verify that before bulk migration**, not after: discovering |
| it does not scale once 150 easy docs are written means retrofitting all of them. |
| |
| **Commits.** (1) Migrate `ConcatAddPairs` / `ConcatSubPairs` into spec, docs |
| included; their text differs between 16/32-bit and 64-bit elements. Verify via |
| `cmd/specls`. (2) Migrate the `Grouped` variants, whose docs vary by element size |
| *and* by the 128-bit grouping — two axes, and the real test of the mechanism. (3) |
| `ShiftSaturated`, `ShiftAllRight`, `ShiftRight`, whose docs differ between signed |
| and unsigned elements; `categories.yaml` selects the variant with a `signed:` |
| join key, which this retires. (4) If the mechanism proves insufficient, extend it |
| (conditional doc text is the likely shape) and document the extension in |
| `internal/spec/doc.go`. |
| |
| `concatSelectedConstant*` is harder still — four spec functions for one API name |
| — but it is *unexported*, so it sits outside `specfill`'s scope per §1.7 and is |
| undecided until `compiler-ops`. Reach for it only if `compiler-ops` has landed |
| and you want a sterner test. |
| |
| **Done when** `cmd/specls` reproduces the existing doc text for all three |
| families, or the doc mechanism has been extended so that it can. |
| |
| ### `spec-common` — Spec the common operations |
| **Done:** [x] · **Needs:** `width`, `doc-explore` · **Blocks:** `validate-amd64`, `comments-yaml`, `spec-all` · **Prefers:** `specfill` · **Risk:** `risk-ref-impl-effort` |
| |
| The method set `midway/comments.yaml` documents: its `.common_methods` block |
| less `String` and `Len` — `Add`, `Sub`, `Min`, `Max`, `And`, `Or`, `MulAdd`, the |
| compares, and the rest of the portable core. `SPEC-STATS.txt` tracks it as |
| "portable core", which is this task's completion bar. Highest leverage in the |
| plan — these are the easiest reference implementations to write correctly, they |
| unlock `comments-yaml` (the worst doc divergence, in the widest-audience |
| package), and they are exactly what a new architecture implements first. |
| |
| The file's keys are not all operations: besides the method docs, it documents |
| vector *types*, functions and templates, and carries receiver names keyed to the |
| shared block. Only the methods/functions are in scope here, and this excludes |
| non-spec residue (`String`, etc, §1.7). |
| |
| **Commits.** Start by sampling ~10 operations at random and timing how long it |
| takes to implement them, to calibrate `risk-ref-impl-effort` before committing |
| to the rest. Then one commit per family (arithmetic, bitwise, min/max, compares, |
| multiply-accumulate), each adding the spec functions with docs and bodies and |
| confirming `cmd/specls` output. |
| |
| **Done when** every *method* name in `midway/comments.yaml` other than non-spec |
| residue resolves to a spec function, with a real body, unblocking |
| `comments-yaml`. |
| |
| ### `spec-all` — Bulk spec migration |
| **Done:** [ ] · **Needs:** `spec-common`, `convert-lo`, `mask-bits` · **Prefers:** `specfill`, `fill-enforce` · **Blocks:** `template-docs` · **Risk:** `risk-ref-impl-effort` |
| |
| The remaining exported operations, prioritized by the names produced by more |
| than one generator (SPEC-STATS.txt §1) — highest drift risk, and each migration |
| fixes several files at once. Best-effort bodies are acceptable; `panic("not |
| implemented")` is safe because conformance skips it. A *wrong* body is worse |
| than no body. |
| |
| Two denominators, both useful. Against the API: the `(receiver, name)` pairs |
| SPEC-STATS.txt §1 shows uncovered, which is what the scoreboard tracks. Against |
| the input being retired: `categories.yaml`'s exported operation names, less the |
| ones `spec-common` handles. They are different counts of different things — one |
| `categories.yaml` name can expand to many API names — so quote whichever the |
| audience needs and say which. |
| |
| **Why `convert-lo` and `mask-bits` gate this.** Only the Converts and LoadStore |
| families actually depend on them — `convert-lo` may rename `ExtendLo*` / |
| `ConvertLo*`, and `mask-bits` may rename the scalable mask bitmap pair, and |
| renaming after the docs are migrated means doing them twice. If either is still |
| open when this task starts, the other families are unaffected and are the place |
| to begin. |
| |
| **Commits.** One per operation family, mirroring the `ops/*/` grouping. Each |
| spec function gains `//specgen:commutative` where the operation is commutative; |
| the directive already exists and is parsed by `specgen`. Drop any |
| `comments.yaml` entries spec gains as they are covered. Include all regenerated |
| outputs in each commit. Once `gen-docs` has landed, each family's regeneration |
| rewrites that family's docs, so review the diff to ensure all doc comment |
| content is equivalent: any dropped text is either something spec should say or |
| an implementation note missing its prefix. |
| |
| **Done when** `specfill` reports no remaining API declarations uncovered by spec, |
| other than the permanent residue named in §1.7 (`String`, `Len`). |
| |
| ## Track E — Retiring the old inputs |
| |
| Retiring the other documentation sources, then `categories.yaml` itself. |
| |
| ### `doc-triage` — Hand-written doc conflict triage |
| **Done:** [ ] · **Needs:** `specfill` · **Prefers:** `spec-common` · **Blocks:** `handwritten-fill` · **Risk:** `risk-silent-doc-loss` |
| |
| Make it safe for `handwritten-fill` to rewrite hand-written files, by emptying |
| the conflict set first. |
| |
| A conflict is a declaration where spec has a doc *and* the hand-written file has |
| different spec-owned paragraphs (§1.6). Those are the only sites `-w` would |
| overwrite: where spec has no entry, the declaration is left alone. Every |
| conflict therefore resolves one of two ways — the prose says something spec |
| should say (move it into spec), or it is about how a target implements the |
| operation (make it an implementation note by giving it one of §1.6's prefixes, |
| and `specfill` never touches it). Neither outcome loses text, which is what makes |
| `-w` safe afterwards. |
| |
| Run `specfill -diff` over the hand-written files — every doc comment it would |
| rewrite is a conflict — and sort each conflict into one of those two. |
| SPEC-STATS.txt §3 breaks the conflict count down by file, with |
| `simd_emulated.go` dominating and a long tail of `archsimd/slicepart_*.go`, |
| `shuffles_amd64.go`, and `extra_amd64.go`. Some implementation text there has no |
| prefix today — `extra_amd64.go`'s "This method compiles to VPTEST x, x" is a |
| `Performance:` note — and the same pass fixes check (a) violations and note |
| spellings in these files. |
| |
| The `slicepart_*.go` files matter most: they declare the `Load*Part` family spec |
| *already* covers via `LoadZPart`, so their prose conflicts on day one — and one |
| of them documents `LoadUint32x4Part` as loading "a Int32x4". |
| |
| **Done when** `specfill -no-fill-names -diff` is empty and `specfill` reports |
| zero check (a) violations. |
| |
| ### `comments-yaml` — Drop spec-covered docs from `midway/comments.yaml` |
| **Done:** [x] · **Needs:** `gen-docs`, `spec-common` · **Prefers:** `fill-enforce` |
| |
| Every method or function-template entry spec covers is dropped from |
| `comments.yaml`. At the time of this task (after `spec-common`) that is the |
| portable core less non-spec residue (`String`, etc, §1.7), plus the |
| Load/LoadPart/Broadcast templates. Entries spec gains afterwards (in `spec-all`) |
| are dropped when spec gains them. |
| |
| **Commits.** (1) Confirm spec covers the portable core; fill gaps. (2) Delete |
| the covered entries from `comments.yaml`. (3) Run `specfill -w` over |
| `simd/simd_emulated.go` — this overlaps `handwritten-fill`; sequence whichever |
| lands first. |
| |
| **Done when** `comments.yaml` has no entry for any operation spec covers, midway |
| sources those docs through `specfill`, and `simd_stubs.go` and |
| `simd_emulated.go` carry identical spec-derived docs for them. |
| |
| ### `handwritten-fill` — Enable `-w` on hand-written files |
| **Done:** [ ] · **Needs:** `gen-docs`, `doc-triage` · **Risk:** `risk-silent-doc-loss` |
| |
| Turn on doc rewriting for hand-written files, once `doc-triage` has emptied the |
| conflict set — at which point `-w` rewrites only docs that already match what it |
| would write. |
| |
| Two guards, both on `cmd/specfill` only. This does **not** change `gentools`' |
| shared `-w`, which every generator uses: |
| |
| - `specfill -w` refuses to run when the report has any findings, or if there are |
| uncommitted changes, so its diff is the complete story of what it did. |
| - `specfill -w` never runs in CI; CI runs report mode (`fill-enforce`). |
| |
| **Done when** `specfill -w` over the hand-written files is a no-op on a clean |
| tree. |
| |
| ### `template-docs` — Retire generator-supplied API docs |
| **Done:** [ ] · **Needs:** `gen-docs`, `spec-all` · **Blocks:** `cat-rehome` |
| |
| Delete the doc strings embedded in wasmgen's and tmplgen's templates, and the |
| `documentation:` fields on exported `categories.yaml` entries, leaving |
| `specfill` to supply them. Mechanical once `spec-all` has covered the operations |
| those generators emit — but not automatic, because their templates interleave |
| prose with the implementation notes that stay with the generator (§1.6). |
| |
| **Commits.** One per generator, stripping the prose and keeping the notes; one |
| for the `documentation:` fields on exported `categories.yaml` entries. Each |
| includes its regenerated output. Then stop setting `AllowDocRewrite` in the |
| generators, so a generator that emits API prose again fails generation. |
| |
| **Done when** no generator's source contains API prose, no exported |
| `categories.yaml` entry has `documentation:`, regeneration is byte-identical |
| to the previous tree, and the generators run without `AllowDocRewrite`. |
| |
| ### `cat-rehome` — Re-home `categories.yaml` non-doc fields |
| **Done:** [ ] · **Needs:** `template-docs` · **Blocks:** `compiler-ops-impl`, `sig-from-spec` · **Risk:** `risk-cat-unknowns` |
| |
| Empty the exported entries of `categories.yaml` of everything but `go:` and |
| `commutative:`, by sending each remaining field where §1.3's inventory says it |
| belongs. `template-docs` has already removed `documentation:`; this is the rest. |
| Compiler-only entries are untouched, for `compiler-ops-impl`. |
| |
| Expect surprises. §1.3's audit was static, and the unifier's behavior on field |
| combinations was not exhaustively traced — `regexpTag`'s role is still |
| unverified (§1.3). Mitigating factor: `go run . -diff` makes every step |
| verifiable byte-for-byte against checked-in generated output. |
| |
| **Commits.** Follow §1.3's disposition column, one field per commit, confirming |
| generated output is unchanged at each step: (1) delete `regexpTag` and `signed` |
| from `categories.yaml` and from `go_*.yaml`, expecting most `go_*.yaml` uses to |
| be inert already; (2) move `constImm` into `go_amd64.yaml`; (3) convert the |
| `out`/`elemBits`/`bits` entries into spec function splits, which `spec-all` |
| should already have done — verify. |
| |
| **Done when** every exported entry in `categories.yaml` has only `go:` and |
| `commutative:` keys; compiler-only entries are untouched, for |
| `compiler-ops-impl`. Generated output is byte-identical throughout. |
| |
| ### `compiler-ops-impl` — Implement compiler-only ops representation |
| **Done:** [ ] · **Needs:** `compiler-ops`, `cat-rehome` · **Blocks:** `sig-from-spec` |
| |
| Implement whatever `compiler-ops` decided, for the compiler-only entries — |
| those that produce no exported declaration — and the `noTypes` / `noGenericOps` |
| switches. No design work should remain at this point; if it does, `compiler-ops` |
| was not finished. |
| |
| These operations have no API surface — they exist only as SSA ops and lowering |
| rules — so nothing here touches `specfill`, spec, or the public packages. It is |
| entirely about giving simdgen somewhere other than `categories.yaml` to read them |
| from. |
| |
| **Done when** no compiler-only entries remain in `categories.yaml` |
| (SPEC-STATS.txt §4) and generated output is byte-identical. |
| |
| ### `cat-delete` — Delete `categories.yaml` |
| **Done:** [ ] · **Needs:** `sig-from-spec` |
| |
| Delete `ops/*/categories.yaml`, the `!import` stub that loads it, and its |
| handling in `simdgen/main.go`. By this point every field it carried has been |
| re-homed (`cat-rehome`), its compiler-only entries have moved |
| (`compiler-ops-impl`), and its signatures come from spec (`sig-from-spec`), so |
| this should be pure deletion. |
| |
| **Done when** `categories.yaml` no longer exists, simdgen runs without it, and |
| generated output is byte-identical. |
| |
| ## Track F — simdgen internals |
| |
| The signature refactor and, eventually, sourcing signatures from spec. |
| |
| ### `sig-refactor` — simdgen signature refactor |
| **Done:** [ ] · **Needs:** — · **Blocks:** `sig-from-spec` · **Risk:** `risk-sig-refactor-convergence` |
| |
| Make the Go signature a first-class value on `Operation` instead of an emergent |
| property of `shape()` → `classifyOp()` → a template name. Ideally shaped so it is |
| directly comparable to `specgen.Func`'s `Recv`/`In`/`Out`. |
| |
| This collapses the stub templates (§1.1 counts them) to roughly three, because |
| operand permutation and parameter naming become data rather than template |
| identity; the `_21`, `_231Type1`, `_31` suffixes become an explicit permutation. |
| |
| It is also the empirical test of whether the unification approach can be kept. |
| The refactor assumes the signature is expressible as data without a new pile of |
| special cases; the quirk fields and `simdgen only supports…` panics catalogued |
| in §1.1 are mostly *upstream* of the signature (XED inconsistency) or |
| *downstream* (register shapes, rules), so the seam looks thin — but that is a |
| prior, not a measurement. If it does not converge, that is the evidence for |
| revisiting §1.1, earned cheaply on the few hundred lines §1.1's layer table |
| isolates. |
| |
| **Commits.** (1) Introduce the signature type and a `GoSignature()` method |
| covering the simplest classes (`op1`, `op2`, `op3`). (2) Extend to immediate |
| variants (`op*Imm*`), which may be fiddly. (3) Extend to permuted and |
| `VecAsScalar` variants. (4) Rewrite the stub templates to consume the signature |
| value; delete the redundant templates. |
| |
| **Done when** `go run . -diff` shows byte-identical output at every commit. |
| |
| ### `sig-from-spec` — simdgen sources signatures from spec |
| **Done:** [ ] · **Needs:** `sig-refactor`, `cat-rehome`, `compiler-ops-impl` · **Blocks:** `cat-delete` |
| |
| simdgen looks up the spec function by `(recv, name)` and asserts signature |
| equality, reporting a structured diff on mismatch. **Key the lookup on |
| `(recv, name)` only** — spec's own uniqueness key — and assert the signature |
| separately. Keying on the signature makes a divergence indistinguishable from |
| "not migrated yet", which is the failure this whole effort exists to catch. |
| |
| This is where `sig-refactor` pays off: the lookup compares against the signature |
| value that refactor introduced, rather than against a rendered template. |
| |
| **Commits.** (1) Look up the spec function and assert equality, failing the |
| generator run on mismatch. (2) Delete whatever signature information in |
| `go_GOARCH.yaml` is now redundant. (3) Switch simdgen to reading commutativity |
| from spec's `//specgen:commutative` directive and delete `commutative:` from |
| exported `categories.yaml` entries. |
| |
| **Done when** simdgen derives every exported signature it emits from spec, |
| commutativity comes from spec, and generated output is byte-identical. |
| |
| --- |
| |
| # Part 4 — Risks and unknowns |
| |
| Every risk is owned by a task, and its substance is stated there. This is the |
| index. `risk-concrete-width`, `risk-impl-defined-behavior` and |
| `risk-ref-impl-effort` are the expensive ones: each reaches every spec body, so |
| each is far cheaper to settle before the bodies are written than after. |
| |
| | | Risk | What it is | Owner | |
| |---|---|---|---| |
| | **[ ]** | `risk-spec-defects` | Known spec defects | `convert-lo`, `mask-bits` | |
| | **[x]** | `risk-concrete-width` | Executing spec at a concrete width | `width` | |
| | **[ ]** | `risk-impl-defined-behavior` | Implementation-defined behavior | `impl-defined` | |
| | **[x]** | `risk-doc-template-ceiling` | Doc-template ceiling — the fallback is a mechanism change affecting every doc written so far | `doc-explore` | |
| | **[ ]** | `risk-ref-impl-effort` | Reference-implementation effort — every migrated operation needs a body that is *correct*, because conformance tests against it; the distribution of difficulty is unknown | `spec-common`, `spec-all` | |
| | **[x]** | `risk-conformance-env` | Conformance execution environment — can CI run arm64/SVE/wasm, and at what cost? | `ci-audit` | |
| | **[ ]** | `risk-sig-refactor-convergence` | `sig-refactor` may not converge — if it does not, §1.1 needs revisiting | `sig-refactor` | |
| | **[ ]** | `risk-silent-doc-loss` | Silent doc loss — under §1.6, text without a note prefix is spec-owned, so implementation text nobody prefixed is dropped on the first rewrite; the guards are review of every rewrite's `-diff` and, once generators emit no prose of their own, `AllowDocRewrite` | `gen-docs`, `doc-triage`, `handwritten-fill` | |
| | **[ ]** | `risk-cat-unknowns` | Unknown unknowns in `categories.yaml` — the field audit was static | `cat-rehome` | |
| |
| --- |
| |
| # Appendix A — Regenerating the two generated files |
| |
| ## `SPEC-STATS.txt` |
| |
| `cmd/specstats` generates [SPEC-STATS.txt](SPEC-STATS.txt), the transition |
| scoreboard: |
| |
| ```sh |
| cd $GOROOT/src/simd/archsimd/_gen |
| go run . -w # (if necessary) ensure generator outputs are up-to-date |
| go run ./cmd/specstats # print the report |
| go run ./cmd/specstats -diff # see what's changed |
| go run ./cmd/specstats -w # rewrite SPEC-STATS.txt in place |
| ``` |
| |
| It loads `simd` and `archsimd` through `go/packages` for each build |
| configuration, taking syntax only — no type-checking, per §1.4 — and reads |
| `simd/internal/spec` through `specgen.Load`. It needs a recent Go toolchain and |
| up-to-date output from the SIMD generators, but does not itself require any ISA |
| data. Commit the regenerated file alongside whatever change moved the numbers; |
| the diff is the record of what moved. |
| |
| It reports only figures that measure how far along the transition is: every one |
| is a fraction heading for 100% or a count heading for zero, tagged with the |
| tasks that move it; indented, untagged numbers are context breaking down the |
| figure above them. If you add a figure, give it a denominator and a direction, |
| or leave it out. |
| |
| One thing it deliberately does not do: it classifies divergence (§5) but fails |
| on nothing; enforcement is `specfill`'s job. `specfill` should absorb the |
| comparison half of this program rather than leave two implementations to drift. |
| |
| For one-off questions the report does not answer, `cmd/specls` prints the API |
| spec currently expands to: |
| |
| ```sh |
| go run ./cmd/specls # the whole expansion |
| go run ./cmd/specls -f Add # one spec function |
| go run ./cmd/specls -f Add -trace # why it expands the way it does |
| ``` |
| |
| ## `SPEC-TASKS.dot` |
| |
| The task graph is drawn from Part 2's two tables by `cmd/spectasks`: |
| |
| ```sh |
| go run ./cmd/spectasks # print the graph |
| go run ./cmd/spectasks -w # rewrite SPEC-TASKS.dot in place |
| dot -Tsvg SPEC-TASKS.dot -o /tmp/tasks.svg # render to SVG |
| go run ./cmd/spectasks | grep -v ^-- | xdot - # view the current graph |
| ``` |
| |
| It reads nothing but this file. Two things it does beyond drawing: |
| |
| It **checks**, from `go test ./cmd/spectasks`. Dependencies are stated three |
| times here — Part 2's table, Part 3's `Needs:`/`Prefers:` lines, and Part 4's |
| `Blocks:` lines, which are the same edges reversed — and the test fails if the |
| three disagree, if the table is not topologically sorted, if a task is in no |
| track or in two, or if an edge points at a task that does not exist. A second |
| test fails if SPEC-TASKS.dot no longer matches the document, so an edit that |
| skips the regeneration step is caught too. The command itself refuses to write a |
| graph from an inconsistent document, but the test is where the problems are |
| reported. |
| |
| It **derives the critical path** rather than restating one: a task is on it when |
| its longest chain in plus its longest chain out spans the whole graph. That |
| keeps every longest chain, not one picked by hand, so parallel roots of equal |
| depth are all marked. |