hoist_invariants: decide invariance modulo the RVar bounds by alexreinking · Pull Request #9521 · halide/Halide

alexreinking · 2026-10-06T19:33:52Z

Stage::hoist_invariants() decided invariance syntactically, so per-block factors in blocked reductions were never hoisted:

out() += cast<float>(data(r)) * scale(r / 32);
out.update().split(r, ko, ki, 32);
Func blocks = out.update().rfactor(ko, u);
blocks.update().hoist_invariants();  // user error before this PR

The intermediate reads scale(promise_clamped(u*32 + ki, ...) / 32), which is scale(u) for every ki in [0, 32).

Now a leaf that mentions RVars has its integer (never float) RVar-dependent subexpressions simplified under the RVars' constant bounds, and is hoisted if the result is RVar-free. rfactor's internal promise_clamped is dropped for this, along with the likely() tag on its value; user promises are kept. When the RVar bounds aren't enough (e.g. a lane var that was an RVar before rfactor), constant bound()s on the Func's pure vars are used too, and the factor is wrapped in a require() so that widening the bound() later fails loudly instead of silently using a wrong factor.

Depends on #9520 for the per-lane case, scale((lane*4 + ki) / 32).

Tests: new cases in correctness_hoist_invariants.

Breaking changes

No API changes. Some pipelines that used to error now hoist, and pipelines that already hoisted may hoist more leaves, which can change float rounding (as hoisting already does).

Authored by GitHub Copilot

kernels/matmul.cpp: out(n, m) = sum_k W(k, n) * X(k, m) in f32, each operand approximated by its format's scheme and severed at its records. kernels/schedule.h: vec_dot for two block-quantized operands is rfactor -> eager_inline -> hoist_invariants -> change_type(Int(32)) -> lanes, which lowers to GGML's ggml_vec_dot_q4_0_q8_0 Neon loop (sdot, per-lane f32 scale-accumulate, two blocks per iteration); otherwise decode + FMA. Formats gain values per record (schemes.h) and acts (formats.cmake); the kernel ABI is struct-typed records [K / block, N | M] whose types come from the kernel metadata; the vec_dot adapter reuses static buffers. q8_0 vec_dot comes with zero kernel changes. Needs the framework's FR-1 (rank-polymorphic BlockReshape/StructLayout, contextual signature) and hoist_invariants modulo RVar bounds (#9521). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hoist_invariants() takes an explicit constant bound on a pure var as its range (#9521); the rfactor'd lane is the in-block vector lane, extent vector size / dot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-10-07T00:49:32Z

Codecov Report

❌ Patch coverage is 91.89189% with 6 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (alexreinking/multiramp-alignment-stacked@5addfc1). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/Func.cpp	91.89%	1 Missing and 5 partials ⚠️

Additional details and impacted files

@@                             Coverage Diff                             @@
##             alexreinking/multiramp-alignment-stacked    #9521   +/-   ##
===========================================================================
  Coverage                                            ?   72.23%           
===========================================================================
  Files                                               ?      262           
  Lines                                               ?    81949           
  Branches                                            ?    20026           
===========================================================================
  Hits                                                ?    59193           
  Misses                                              ?    16559           
  Partials                                            ?     6197

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.