[kylebernhardy]

⊙ Problem

Applications can require text matches or combine alternatives, but cannot reward an extra match without changing eligibility. Index field weights also cannot express a boost for one request.

Refs Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search.

🤔 Is this the right first slice of query relevance controls?

Deliver boosts and required-plus-optional scoring first. These compose with existing text modes without rebuilding indexes. Regex, similar-document retrieval and a separate per-request field-weight map remain outside this slice; the linked issue stays open.

💡 Solution

Draft / release gate: reconcile the parallel telemetry ABI contract and choose an unused synchronized package version before publication. No release is performed by this PR.

• boost multiplies a subtree's score while preserving its matches.
• optional requires one expression and adds scores from matching optional expressions. Optional-only matches never enter the result.
• Boost a field-restricted clause to adjust its contribution for one request; configured index weights still apply.

{
  query: {
    operator: 'optional',
    required: { text: 'waterproof', fields: ['title', 'description'] },
    optional: [{
      operator: 'boost',
      boost: 3,
      clause: { text: 'trail running', mode: 'phrase', fields: ['title'] }
    }]
  }
}

⚖️ Alternatives

• Existing OR queries: admit optional-only matches, changing the required result set.
• Index field weights: apply across requests and do not express a boosted phrase or arbitrary subtree.
• A custom scorer: would duplicate Tantivy's Boolean scoring and increase the risk of search/paging score drift. Native query composition keeps those semantics together.

🔧 Changes

Expression flow and compatibility

From request to result

The request keeps its required membership while optional matches contribute to ranking.

flowchart LR
  API[Typed expression] --> TS[Normalize and bound]
  TS --> Wire[Packed tags 4 and 5]
  Wire --> Rust[Validate and classify]
  Rust --> Native[Tantivy query tree]
  Native --> Results[Ranked hits and counts]

Loading

ts/native.ts exposes both expression variants, normalizes boosts once to float32, rejects conflicting shapes, and carries the normalized expression through search and continuation identity. Optional normalization captures an array length once for validation, budget charging and allocation, preventing Proxy-backed lists from expanding uncharged work. ts/codec.ts appends scoring tags while retaining tags 0–3 for existing leaves and Boolean groups.

src/protocol.rs decodes the new variants and accounts for their structure and admission class. src/request_validation.rs also validates direct Rust requests. Boosts must round to finite float32 values in 0..=100, excluding negative zero; optional lists must be nonempty. The shared native Optional-child check gives the Optional-specific diagnostic in typed validation, wire admission and decoding, while retaining Boolean diagnostics and error codes. Existing depth, byte, clause and deadline bounds apply to every child.

src/lib.rs advertises native ABI 14 and query API 8; ts/load-addon.ts requires that matching addon. This changes request compatibility but not persisted index identity. The parallel Expose fulltext operation telemetry and latency histograms, verified at 4840b32a, also claims ABI 14 but query API 7 plus telemetry API 1. These incompatible contracts must be reconciled across native exports, runtime info, facade, loader, tests and docs before release. If telemetry ABI 14 ships first, scoring requires a fresh ABI and Harper compatibility-pair update; otherwise combine both unreleased contracts into one rebuilt artifact.

Native scoring and preserved invariants

⚠️ Look hardest: src/engine.rs builds optional scoring as one native Must branch plus native Should bonuses. The required branch alone decides eligibility for filters, candidate restrictions, counts and score/ID pagination. Only the required branch supplies a positive admission anchor; optional branches still contribute their expensive-work classification.

Only root boosts over Any/All leaves multiply selected field weights before building term queries, preserving the query topology used by native pruning. Recursive boosts retain native BoostQuery multiplication order: an inner large field weight must not overflow before an enclosing zero boost reaches it. Other root modes and complex subtrees also use native BoostQuery composition. At the root, this wrapper loses specialized Boolean pruning. Optional scoring over required All also adds work over the required-only intersection. Nested boosts multiply without a separate cumulative cap. Nonfinite result scores fail explicitly. Zero boosts preserve membership.

Compiler-owned stable routing travels in BuiltQuery.stable_scores. Term, Phrase, Fuzzy, prefix and Boolean builders determine it from the query they actually construct, replacing the duplicate AST-mode walk and division heuristic. Completed prefix/fuzzy-prefix and singleton exact-prefix paths can lower to the same faulty native conjunction; empty per-segment fields can also collapse unions into bare term scorers. Non-unit native Boost wrappers use generic scoring and clear the stable-routing flag; identity boosts inherit it, while candidate/filter wrappers keep their existing behavior. A four-term native canary compares raw TopDocs with scorer iteration. Stable collection scans matches and resolves IDs beyond the visible page. Both ordinary stable passes now reuse the shared deadline collector in src/catalog.rs, including the boundary-tie rerun. Its deletion-aware scorer loop checks the shared deadline every 1,024 visited documents and rejects the whole request on expiry. The engine error mapping preserves E_TIMEOUT; no partial hits or cursor escape. These cooperative checks do not preempt a slow individual scorer operation. src/DESIGN.md records remaining cost and native-bound limits; this is not a universal pruning repair or strict latency guarantee.

Callers must authorize optional fields as well as required fields. An omitted field list searches every configured text field. traceMatches() remains a simple-clause tracing API; boosts do not change spans.

Evidence and caller guidance

test/query-scoring.test.mjs covers scoring, eligibility, restrictions, filtered count fallback, paging, validation and wire tags. Native engine/protocol/request tests cover typed Rust paths and scorer topology. New public compiled-query regressions compare bounded search with continuation scores at a distinct cutoff, including multiple segments, empty secondary fields, completed-prefix constructors and real candidate/filter restrictions. All and AND restriction controls each admit 150 of 300 documents. Two-term boosted All controls retain the native pruning path. The early-abort regression fails before the deadline fix by visiting all 4,096 matches; the fixed path aborts within 1,024 advances. Its deadline-enabled deletion control also preserves exact score bits and excludes tombstoned documents. Independent required-plus-two-bonus arithmetic rejects max-only scoring; a 127-record, varied-addend paging fixture catches real score-bit drift when stable routing is removed. Native nested-boost regressions fail before the fix for fractional weights and a large finite field weight under an outer zero boost. test/native.test.mjs and test/load-addon-resolution.test.mjs expect the new capabilities.

benchmarks/query-scoring.mjs adds single-term and two/three/four-term All baseline, root-lowered and native-wrapped controls. Paired correctness and score-bit checks precede timing; both default tied and varied modes complete 40 rows over four committed segments. The opt-in --routing-shapes mode adds 20 rows per corpus for OR, singleton prefix, short Fuzzy, one-token two-field Phrase and three-term two-field All. Complete traversal reads respect prefix limits; timed requests use limit 20. package.json adds the benchmark command. The benchmark record distinguishes route-specific results from throughput, catalog-scale memory and general pruning guarantees.

README.md and examples/query-modes.mjs show the caller contract; the README limits undefined-as-absent behavior to the listed scoring/structural properties, excluding present text: undefined. docs/limits.md documents Rust enum migration and removal of Eq. DESIGN.md indexes the invariants in src/DESIGN.md.

✅ Verification

Executed qualification results and environment limits

Latest 5394ecf changes only the empty-Optional diagnostic and regression coverage; 163 Rust tests passed, 6 ignored, with Clippy, TypeScript, full formatting and diff checks passing. The regression failed before the fix and now covers all three native entry paths plus unchanged empty AND/OR messages. The scoring implementation remains 2b48620. Previous-artifact Node qualification passed 467 main, 19 metadata and 12 worker tests, optimized scoring/control qualification 52 tests, and clean-environment release qualification 2 packed-consumer and 6 production-worker tests. No addon rebuild or Node-suite rerun is claimed for the diagnostic-only response.

The deadline regression proves early cancellation instead of a post-scan error: it advances no more than 1,024 of 4,096 matching documents. A real deletion with merge disabled verifies live-document filtering, exact total 4,095 and exact score bits against the unwrapped stable collector. Existing four-term and public bounded-page regressions cover demonstrated compiler constructors, multiple segments, candidate restrictions and metadata filtering; they do not qualify every native rounding bound.

The final optimized addon SHA256 is 652d1aaafd045641c1880a6c08e1a52099df5aa599de91fabf64f1ec6191cd6b. Root and installed ABI 14/query API 8 artifacts match, and the ABI-13 backup is preserved. Canonical npm run test:node reached the packed phase but npm 12 injected EALLOWSCRIPTS; the direct canonical release runner passed after removing inherited allow-scripts variables. User npm configuration was unchanged.

On this artifact, the unchanged default benchmark completed 40 rows per tied/varied mode, and opt-in routing shapes completed 20 rows per mode, each observing four committed segments. Varied exact-total-off p50 baseline/root/native-wrapped was All2 0.077/0.078/0.102 ms, All3 0.163/0.162/0.127 ms and All4 0.177/0.177/0.143 ms. Routing baseline/native-wrapper was OR 0.152/0.110 ms, prefix 0.159/0.129 ms, short Fuzzy 0.156/0.112 ms, Phrase over two fields 0.156/0.114 ms and All3 over two fields 0.251/0.227 ms. Membership, totals, scaled scores and distinct-boundary checks passed. These warm total-request measurements do not isolate pruning or deadline overhead and establish no general speedup or catalog-scale bound.

Harper's final paired deadline-aware 652d artifact passed 113 focused and 20 deployed tests, with the actual resolved addon hash verified. Local evidence: query-relevance-harper-deadline-final-focused.log and query-relevance-harper-deadline-final-deployed.log. Hosted scoring CI remains gated on wrapper release and the Harper dependency pin/lock update.

Persisted-index interoperability previously passed with real addons: ABI 13 seeded a document, ABI 14 reopened and added another, and ABI 13 reopened and read both. No persisted identity change or index rebuild is introduced.

Review response and stopping point

Five whole reviews preceded the focused Claude/domain delta at 5394ecf, which resolved the Optional diagnostic and earlier measurement findings. Final 48e0fbf retains the historical benchmark baseline; its values were parent-checked without another outside review. Gemini authentication remains unavailable.

The remaining package-version finding is a release gate. Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search tracks native pruning cost, strict-property compatibility, the tied diagnostic mode and the telemetry collision. All five wrapper deferral markers are filed there. Its open issue was checked directly, without buying another code review solely to refresh bookkeeping.

Coordinated guide: Document full-text boosts and optional scoring, local head c7e9a5b803aece59e702b20811757c1c514b50fa. Harper integration: Add full-text boosts and optional scoring conditions, local head a53592d9fa62df12d5a1366c5e948e188884bb78.

Comment generated by kAIle (GPT-6)

Related PRs: #81 overlaps (verified ABI14/queryAPI7/telemetryAPI1 collision; reconcile with scoring14/8 before release: Expose fulltext operation telemetry and latency histograms), #79 overlaps (updates published-version guidance from Prepare fulltext 0.8.0 for release), #78 overlaps (extends request validation and admission from Expose fuzzy matching controls and align Rust request validation), #77 overlaps (extends structural budget paths from Make analyzed query capacity configurable per handle)

Complexity: complicated

Historical review context

The initial Grok findings below are superseded by the final response review at 2c3edf9: shape validation, native multiplication order, parity and benchmark evidence are addressed. Composite pruning remains an accepted measured tradeoff tracked in Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search. The historical managed block is retained unchanged.

Late Grok findings

4 surviving (highest major) — Grok reviewed 3db8a90, ruled against 358e1bb

• major — ts/native.ts:1459 — boost and optional expressions silently discard sibling clause/clauses properties, so intended constraints never reach native search
• major — src/engine.rs:1512 — boosts around composite expressions still hide BooleanWeight pruning behind BoostQuery, making low-limit searches scan every match
• minor — README.md:574 — the upgrade guide advertises query API 7/native ABI 13 although this build requires 8/14, leading operators toward incompatible artifacts
• nit — ts/codec.ts:1098 — zero-accepting float validation retains an incorrect error message, alongside object stringification and compressed design prose

Not standing: 6 resolved. Grok ran 17m (17m from launch to report); the fold adjudication (cursor-grok@3db8a904fce8-8710a551, codex) took 2m.

Deferred findings

• 3084f89c82b555f2 none performance — src/engine.rs — Composite boosts bypass Tantivy specialized TopDocs pruning — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e313f09f9410c6e0 none correctness — README.md — Boolean groups reject leaf-only fields and mode properties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e33b81c1db8c0e17 none performance — benchmarks/query-scoring.mjs — Default benchmark corpus retains score-boundary ties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• e5b36a05fc329c13 none correctness — ts/native.ts — Boolean groups reject formerly ignored leaf-only properties — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55
• 8703f3426d90022d none reliability — package.json — ABI-14 source reuses the published 0.8.0 artifact coordinate — filed Extend typed queries with optional scoring clauses, boosts, regex, and similar-document search #55

Late Grok findings

5 surviving (highest blocker) — Grok reviewed 5d809c9

• blocker — package.json:75 — a clean install resolves published ABI-13 native packages for an ABI-14 facade and fails addon loading
• major — benchmarks/query-scoring.mjs:25 — the hot-path root-leaf boost specialization lacks a benchmark that exercises its specialized-pruning purpose
• minor — test/query-scoring.test.mjs:344 — root-boosted single-field All lacks public search-versus-traversal score parity coverage
• nit — src/protocol.rs:1311 — empty Optional frames report a Boolean-specific diagnostic through all native entry paths
• nit — README.md:510 — the undefined-property rule overstates behavior because present text: undefined still rejects

Not standing: none. Grok ran 19m (19m from launch to report); the fold adjudication (cursor-grok@5d809c96ad57-b4c84b62, codex) took 2m.

_{Review-Coverage: authored=codex; ran=cursor-composer,claude,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; blocked=gemini(auth); declined=muse; rounds=9; full=5 @ 48e0fbf}

_{Review-Attention: deep ~25m (critical: codec.ts; decisions: do-less-alternative, stable-score-routing, conservative-three-term-cutoff, abi-without-version-bump) @ 48e0fbf}