Skip to content

fix(coverage): fix CUDA graph test timeouts in Windows coverage CI - #3199

Draft
rluo8 wants to merge 2 commits into
NVIDIA:mainfrom
rluo8:fix/coverage-graph-main-thread
Draft

rluo8 wants to merge 2 commits into
NVIDIA:mainfrom
rluo8:fix/coverage-graph-main-thread

Conversation

@rluo8

@rluo8 rluo8 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Description

Windows coverage tests currently run pytest on a worker thread to provide an 8 MB stack. CUDA graph cleanup uses Python pending calls, which require the main thread; this can cause graph tests to time out.

This PR runs pytest on the main thread while retaining the larger stack. On Windows, the helper launches a temporary copy of the base Python executable with an 8 MB stack reservation and preserves the virtual environment.

@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Oct 23, 2026
@rluo8 rluo8 changed the title fix(coverage): Run Windows coverage tests on the Python main thread fix(coverage): fix CUDA graph test timeouts in Windows coverage CI Oct 23, 2026
@rluo8
rluo8 marked this pull request as ready for review October 23, 2026 08:13
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

📝 Summary

Summary by CodeRabbit

  • Chores
    • Automated tests now support larger main-thread stacks across Windows and other platforms, helping accommodate tests with higher stack requirements.
    • Deployment now runs only after the required job succeeds and only for the NVIDIA repository’s main branch. Other branches and refs no longer trigger deployment.

The pytest wrapper now runs tests on the main thread and adjusts or checks the requested stack size. The coverage workflow comment describes the Windows stack settings, and its deploy job now runs only for successful jobs on the main branch in the NVIDIA repository.

Changes: Pytest main-thread stack

Layer / File(s) Summary
Prepare and relaunch the Windows interpreter
ci/tools/run_pytest_with_stack.py
The wrapper validates and updates the interpreter’s PE stack reserve, then re-executes itself with the requested stack size and pytest arguments.
Check the main-thread stack and run pytest
ci/tools/run_pytest_with_stack.py, .github/workflows/coverage.yml
The wrapper checks or raises the main-thread stack, validates the requested size, and runs pytest on the main thread. The coverage comment describes the 1 MB default and 8 MB helper stack.

Coverage deployment condition

Layer / File(s) Summary
Restrict coverage deployment to main
.github/workflows/coverage.yml
The deploy condition now requires refs/heads/main in addition to the existing owner and job-success conditions.

Possibly related issues

  • NVIDIA/cuda-python issue 2722: The changes address one coverage failure mode by running pytest on the main thread with an 8 MB stack.

Suggested reviewers: jpascucci-nv, rwgk

Priority: ➖ Normal

Change: Bug fix

Merge Risk: 🔵 Low · up to 9a73f

Some unsupported stack-size requests fail with a traceback instead of a clear error. This is a bounded issue to fix or accept before merging.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between afa6249 and 9a73fa8.

📒 Files selected for processing: 2
  • .github/workflows/coverage.yml
  • ci/tools/run_pytest_with_stack.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.


soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
if soft != resource.RLIM_INFINITY and soft < stack_bytes:
resource.setrlimit(resource.RLIMIT_STACK, (stack_bytes, hard))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '1,135p' ci/tools/run_pytest_with_stack.py
rg -n 'run_pytest_with_stack|stack-mb|RLIMIT_STACK' .github ci tests

Repository: NVIDIA/cuda-python

Length of output: 5595


🌐 Web query:

official Python documentation resource.setrlimit ValueError soft limit exceeds hard limit

💡 Result:

<source_evidence>
<source kind="search">
<title>resource — Resource usage information — Python 3.14.7 documentation</title>
<location>https://docs.python.org/3/library/resource.html</location>
<excerpt>citeturn0search0 [wordlim: 200] Crawled: last month; The soft limit can never exceed the hard limit. ... resource.setrlimit(resource, limits)¶ ... Raises `ValueError` if an invalid resource is specified, if the new soft limit exceeds the hard limit, or if a process tries to raise its hard limit.
</excerpt>
</source>
<source kind="search">
<title>resource — Resource usage information — Python 3.15.1 documentation</title>
<location>https://docs.python.org/3.15/library/resource.html</location>
<excerpt>citeturn0search1 [wordlim: 200] Crawled: yesterday; resource.setrlimit(resource, limits)¶ ... Raises `ValueError` if an invalid resource is specified, if the new soft limit exceeds the hard limit, or if a process tries to raise its hard limit.
</excerpt>
</source>
<source kind="search">
<title>resource — Resource usage information — Python 3.14.7 documentation</title>
<location>https://docs.python.org/uk/3/library/resource.html</location>
<excerpt>citeturn0search2 [wordlim: 200] Crawled: last month; resource.setrlimit(resource, limits)¶ ... Raises `ValueError` if an invalid resource is specified, if the new soft limit exceeds the hard limit, or if a process tries to raise its hard limit.
</excerpt>
</source>
<source kind="search">
<title>resource — Resource usage information — Python 3.15.0 documentation</title>
<location>https://docs.python.org/3/library/resource.html?highlight=option</location>
<excerpt>citeturn0search3 [wordlim: 200] Crawled: 4 days ago; Each resource is controlled by a pair of limits: a soft limit and a hard limit. ... The soft limit can never exceed the hard limit. ... Raises `ValueError` if an invalid resource is specified, or `error` if the underlying system call fails unexpectedly.resource.setrlimit(resource, limits)¶
</excerpt>
</source>
<source kind="search">
<title>cpython/Doc/library/resource.rst at main · python/cpython · GitHub</title>
<location>https://github.com/python/cpython/blob/main/Doc/library/resource.rst?plain=1</location>
<excerpt>citeturn0search4 [wordlim: 200] Crawled: 4 days ago; Each resource is controlled by a pair of limits: a soft limit and a hard ... Raises :exc:`ValueError` if an invalid resource is specified, or ... .. function:: setrlimit(resource, limits)
</excerpt>
</source>
<omitted_sources count="26">Additional source records were omitted from this bounded result.</omitted_sources>
</source_evidence>

Citations:

- 1: https://docs.python.org/3/library/resource.html?utm_source=openai
- 2: https://docs.python.org/3/library/resource.html?utm_source=openai

suggestion: Reject requests above the hard limit.

resource.setrlimit raises ValueError when the requested soft limit exceeds a finite hard limit. Silently clamping would violate the wrapper's contract because it would not provide the requested stack.

--- "a/ci/tools/run_pytest_with_stack.py"
+++ "b/ci/tools/run_pytest_with_stack.py"
@@ -83,9 +83,14 @@
     else:
         import resource
 
         soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
         if soft != resource.RLIM_INFINITY and soft < stack_bytes:
+            if hard != resource.RLIM_INFINITY and stack_bytes > hard:
+                raise RuntimeError(
+                    f"Cannot ensure requested stack size: {stack_bytes} exceeds "
+                    f"RLIMIT_STACK hard limit {hard}"
+                )
             resource.setrlimit(resource.RLIMIT_STACK, (stack_bytes, hard))
 
 
 def main():
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
resource.setrlimit(resource.RLIMIT_STACK, (stack_bytes, hard))
if hard != resource.RLIM_INFINITY and stack_bytes > hard:
raise RuntimeError(
f"Cannot ensure requested stack size: {stack_bytes} exceeds "
f"RLIMIT_STACK hard limit {hard}"
)
resource.setrlimit(resource.RLIMIT_STACK, (stack_bytes, hard))

@rluo8
rluo8 marked this pull request as draft October 23, 2026 09:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant