Skip to content

Skip lookup rows where every lookup is disabled - #19

Merged
0xddom merged 201 commits into
mainfrom
shankara/skip-vacuous-lookup-rows
Oct 16, 2026
Merged

0xddom merged 201 commits into
mainfrom
shankara/skip-vacuous-lookup-rows

Conversation

@shankarapailoor

@shankarapailoor shankarapailoor commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

codegen_lookup_invocations clones the lookup template onto every row of a group and only then binds it to the row. Rows where every lookup selector is off still get a full copy, which is folded away much later. For circuits with large table regions this dominates memory: on the packed SHA3 circuit (sha3/digest_1/sha3/byte), IR generation ran out of memory (>22 GB). Its 12,287-row spread table region alone produced 3.9M lookup statements, none of them live.

Gates already skip rows whose selectors are disabled; lookups did not.

Change

Before cloning, compute for each lookup the selectors that multiply every input (find_selector_factors), and skip a row when every lookup has such a selector disabled on it. In that case all inputs of every lookup are zero, so each lookup only checks the all-zero tuple.

The check is conservative:

  • a selector gates an expression only if it is a factor of the whole expression (a * b → union of factors, a + b → intersection), so q * a + (1 - q) * b is never skipped;
  • a lookup without a gating selector is kept on every row.

Skipping only removes constraints, so the IR can only gain spurious solutions, never lose real ones. If a table lacked the all-zero tuple, disabled rows would be unsatisfiable in the circuit, and dropping them can only cause a false alarm downstream.

Results (SHA3 harness)

before after
rows instantiated 16,480 3,428
IR generation OOM (>22 GB) 8 s, ~2.3 GB peak

Testing

  • New unit tests for find_selector_factors (products, sums, negation/scaling, q * a + (1 - q) * b).
  • cargo test -p haloumi-ir-gen --all-features passes.
  • End to end on the SHA3 extraction: the kept rows match an independent per-row count of enabled lookup selectors (3,386 / 8 / 34 per group), and extraction to LLZK completes. I did not diff the output against an unfiltered run, since that run does not fit in memory.

Notes

  • cargo clippy -p haloumi-ir-gen -- -Dwarnings and cargo +nightly fmt --check already fail on dani/cargo-haloumi, for code this PR does not touch (gates/rewrite.rs:104, ctx.rs:139, patterns.rs:86, and 12 formatting diffs). The files changed here are clippy- and fmt-clean.
  • The gate filter in FallbackGateRewriter uses find_selectors (any selector mentioned) together with gate_is_disabled, so it may skip a gate like q * a + (1 - q) * b on rows where q is off, and a gate with no selector on every row. That's not addressed here; find_selector_factors could be reused for it.

🤖 Generated with Claude Code

0xddom
0xddom previously approved these changes Oct 2, 2026

@0xddom 0xddom left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I'm rebasing this one on top of #20 for easier stacking

@0xddom
0xddom changed the base branch from dani/cargo-haloumi to shankara/llzk-non-builtin-field October 2, 2026 10:07
0xddom and others added 17 commits October 13, 2026 20:00
The runner always passed --llzk-field-name to LLZK as a builtin field, so
circuits over other primes could not be emitted and a mislabeled field was
silently accepted. For example, a circuit over BLS12-381's scalar field
labeled `bn254` produced LLZK over BN254's prime.

Builtin names are now checked against the circuit's prime and rejected on
a mismatch. Any other name defines a field with the circuit's prime.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@0xddom
0xddom force-pushed the shankara/llzk-non-builtin-field branch from f19a753 to f42ac56 Compare October 13, 2026 18:00
0xddom and others added 6 commits October 13, 2026 20:01
Lookup IR was instantiated on every row of a group, including rows where
all lookup selectors are off, such as the rows of table regions. Those
copies fold away later, but only after being fully materialized; on the
packed SHA3 circuit IR generation ran out of memory (>22 GB).

A lookup is now vacuous on a row when a selector multiplying every one of
its inputs is disabled there, and rows where every lookup is vacuous are
skipped. A selector that only appears inside a sum, as in
`q * a + (1 - q) * b`, does not gate the lookup, and lookups without a
gating selector are kept on every row.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@0xddom
0xddom force-pushed the shankara/skip-vacuous-lookup-rows branch from 36e5db0 to 3e78151 Compare October 13, 2026 18:01
Base automatically changed from shankara/llzk-non-builtin-field to main October 16, 2026 11:52
@0xddom
0xddom dismissed their stale review October 16, 2026 11:52

The base branch was changed.

@0xddom
0xddom merged commit 13f80e1 into main Oct 16, 2026
6 checks passed
@0xddom
0xddom deleted the shankara/skip-vacuous-lookup-rows branch October 16, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants