Repository navigation
Kernel gaps - #49
Merged
Merged
Kernel gaps#49
Conversation
The policy issue #9 asked for, as the code already does it (tested by test_dot's signed negative values and test_cut_range): the cut is never negative, so a sparse list holds no negative pixels; the powder sums every unmasked pixel as it is, so dark-subtracted CCD negatives integrate, and a Pilatus's gap and bad-pixel markers (-1, -2) belong in the mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
i8, i16 and i32 pixels took the scalar collect whatever the pipeline said. Every SIMD collect (avx512cs, avx512f, avx2cs, sse2, vsx, neon) now takes a template argument S: the same kernel on the same bits with a signed compare (AVX-512 epi compares, no sign bias for SSE2/AVX2, the signed NEON/VSX compares, the scalar tails through bslz4_gt<S>). collect_gt<int8/16/32_t> call them with S; collect_nz<int*> are the unsigned ones on the same bits (!= 0 is > 0 there). The unsigned instances are unchanged. i16 joins u8/u16 in the low-planes values: the cut is never negative, and a negative pixel sets the high planes, so its block takes kcb as a u16 block >= 256 does. i8 stays on kcb (its negative bytes would pass the fused collects' unsigned byte compare). Zen 4, sparsify fps (auto pipeline, before -> after): silicon i16 as recorded / zeroed <= 4 285 -> 769 / 339 -> 1507 silicon i8 as recorded / zeroed <= 4 323 -> 985 / 370 -> 2464 ceo2 i32 as recorded / zeroed <= 150 121 -> 277 / 216 -> 420 (i32 now within 7% of u32; i16 is held back by the high planes the -1 gap pixels set, which lz4 has to decode.) Tests: test_collect_tiers runs every value for i8/i16/i32 with -1 and minimum pixels; test_lowplanes_fused adds i16 frames with -1 pixels. x86 1552 passed; POWER9 1390 passed; aarch64 cross-build clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Correction to 0f7a8bc: older Pilatus detectors change their -1/-2 markers from frame to frame, which a static mask cannot follow. Say so: only always-marked pixels can be masked; moving markers are summed as values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Older Pilatus detectors mark gaps and bad pixels with -1/-2 that change from frame to frame, which a fixed mask cannot follow. chunk2sparse and chunk2sparseCSC (and plan.context) take negatives="drop" (default, as before: CCD frames with the dark subtracted) or "list": for signed integer pixels the pixel list also holds every negative pixel, whatever the cut, and the caller corrects the powder and its normalisation from it afterwards (docs/index.md, "Pixel types and negative values", now describing both cases). The powder is unchanged either way. It costs nothing in the kernels: with a cut >= 0, "v > cut or v < 0" is the unsigned compare on the same bits, so the signed collects run with S = false. The flag rides on the pipeline's collect value (BSLZ4_COLLECT_NEGATIVES, taken off by bslz4_resolve into bslz4_pipe.list_negatives and put on bslz4_work.collect by the driver); the dot paths' list filters and bslz4_cut_above_max honour it; collect_nz drops it before the unsigned dispatch. Refused for unsigned and float pixels (markers in an unsigned container are above any cut already) and with howmany. test_negatives.py: the list against numpy for i8/i16/i32/i64, every usable collect value, low-planes for i16, chunks and from_frames, a cut above the type's maximum; the powder and the documented correction on both routes for csc, csc-run, padded, bsb-csr, csc-nosplit, bsb-csr-nosplit and csc-nosplit-int; the refusals. x86 1561 passed; POWER9 1399 passed; aarch64 cross-build clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f32 pixels (dark-subtracted CCD frames, usually through from_frames) took the scalar collect. The six 32-bit kernels take a second template argument F: the same kernel on the same bits with a float compare (AVX-512 cmp_ps GT_OQ, SSE2/AVX2 cmpgt_ps without the sign bias, vcgtq_f32, the VSX float compares, the scalar tails through bslz4_cmp<S, F>); NaN is never above the cut, as in the scalar loop. collect_gt<float> calls them with F; collect_nz<float> is the u32 one on the bits (every non-zero float, NaN included as before, and -0.0, which adds nothing to the dot and is never above a cut). f64 stays scalar. Zen 4, ceo2 frames as f32 minus a dark of 10, sparsify at cut 150 through from_frames: 355 -> 1002 fps (u32: 991). test_collect_tiers runs every value for f32 with negatives, -0.0, NaN and +-inf and fractional cuts. x86 1571 passed; POWER9 1409 passed; aarch64 cross-build clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.