Repository navigation
Conversation
…issue 43) decompress, integrate and their _from_device forms take out=: any GPU array with __cuda_array_interface__ (cupy, torch, numba, ours) on the object's GPU, C-contiguous, writeable, of the result dtype. It is written and returned; nothing is allocated for the result, so a loop allocates no GPU memory once the object's buffers exist. Without out, unchanged. Example 14 shows out= with cupy and torch, and how to keep one of our arrays (copy, wait for the copy, then drop ours): a dropped array's block goes back to the pool, and the next call may write it while another stream still reads it (checked on an L40S, both backends). jax now runs with x64: without it from_dlpack converted the float64 cake to a float32 copy. Page rebuilt on an L40S with cupy, torch, jax, numba, pycuda and tensorflow. Tests: both backends, 1142 passed, 16 skipped (L40S). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pool gave blocks out and took them back on the legacy default stream, but the GPU classes write their results on their own non-blocking stream. So a block freed while torch's or cupy's default stream still had a read of it queued went straight to the next result, in this thread or another, and was overwritten before that read ran (seen with torch on an L40S). empty() now takes the block on the current stream, and the GPU classes make theirs current first: the pool then waits for the free on the legacy stream before handing the block over. zeros() and asarray(), which work on the legacy stream, still take it there. Readers on streams of their own (torch.cuda.Stream, jax) must still finish before dropping an array, as with cupy's pool. decompress() loop on an L40S, 2162x2068 uint32, 200 calls: 1.62 -> 1.63 ms per call for 8 frames, 0.57 -> 0.57 for one. The new test fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exact: float64 inside jax.experimental.enable_x64, same memory. Fast: an acc="f32" object, float32 sums shared with jax's default float32 (largest relative error 1.5e-7 on the silicon data). Default: our float64 given to plain jax, which makes a float32 copy, rounded, without a word. The copy kept in the ownership section is made under x64. Page rebuilt on an L40S. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In the integrate docstring and example 14. Seen with jax 0.4.35: 3*2**31 + 7 came back as -2147483641. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d.record_stream(stream) names a stream (torch, cupy, numba, ours, or a handle) that still uses the array. When it is dropped, its block is given back on a stream of ours that first waits for the legacy default stream and each recorded stream, so the next result cannot take it while that work is queued. Arrays with no recorded stream are freed as before. The free stream is made with the pool: __del__ takes no lock. torch.cuda.Stream reading our array, dropped at once, next call: the read saw the next call's data without record_stream, the right data with it (L40S). The test (a cupy stream) fails without the call. Example 14 shows it with torch; the decompress docstring says so. decompress() loop unchanged: 1.62 ms per call (8 frames), 0.57 (one). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stream Before writing a caller's out, or reading chunks in a caller's GPU buffer, our stream waits (on the GPU, no host wait) for the legacy default stream: torch's default-stream work on those arrays no longer needs a synchronize first. Both backends. Driver backend: __dlpack__ records the consumer's stream (torch.from_dlpack and cupy.from_dlpack pass their current one), as record_stream does. The cupy backend's device_bytes waits for a cupy array's own stream, as the driver backend's does. Stream.__del__ is quiet at interpreter shutdown. Example 14 shows the automatic case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ities set after allocation group None took every frame of the call up to 256, whatever their size: 256 Eiger 4M u32 frames stage about 1.26 GB of chunks, on the device and pinned on the host, out of memory on a 1 GB Quadro 600 and tight on a 6 GB TITAN Black. Now the largest group up to that whose slab (1.25 x its chunks) takes at most 1/16 of the device memory and of the host memory, at least 1; an explicit group still wins. Chunks already on the device stage nothing: unchanged. _ensure set its capacities before making the buffers they describe: after an out-of-memory error the object believed it had the larger buffers and the next call could write past them. They are set once the buffers exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.