CI: update certificate-related packages for the eMudhra/emSign CA incident by juenglin · Pull Request #3197 · NVIDIA/cuda-python

juenglin · 2026-10-22T23:38:38Z

Raise packages below the fixed versions from the Python security advisory (https://discuss.python.org/t/109576):

ci/pixi-version.env: pixi v0.73.0 -> v0.82.1
pypa/cibuildwheel v4.3.1 -> v4.3.4
pypa/gh-action-pypi-publish v1.14.2 -> v1.14.3
pixi lockfiles (all six workspaces): certifi 2026.10.19, pip 26.2.2, uv 0.13.5 and ca-certificates 2026.10.19, updated per package with pixi 0.82.1 (no general refresh, per the lockfile maintenance pause)
ci/README.md: the pause section no longer claims the pixi pin and lockfiles are unchanged

Refs #3195

Commits

Update certificate-related packages: the pins, ci/README.md, and the root, cuda_pathfinder, cuda_bindings_13 and cuda_core lockfiles, plus ca-certificates in the benchmark lockfiles.
Update uv in the benchmarks source-package host environments: locks uv 0.13.5 (and ca-certificates 2026.10.19) in the host environments of the source packages in benchmarks/cuda_bindings and benchmarks/cuda_core.

Notes for reviewers

uv is not a dependency of any environment; it appears only in the host_packages of the pixi-build source packages, so pixi update uv cannot target it. In each workspace I temporarily added uv = "*" to pixi.toml, updated the source packages, and reverted the manifest. No pixi.toml is changed by this PR.
The benchmark workspaces pin python = "3.14.*" but define no [workspace.build-variants], so re-solving their source packages failed (the re-evaluated packages required python_abi 3.15). The second commit used a temporary python = ["3.14.*"] build variant to get past this, also reverted afterwards.
Re-solving the source-package host environments also moved some unrelated packages: cuda-pathfinder 1.8.4, CUDA 13.4.92 build _1 (including cuda-bindings 13.4.4 in cuda_core), binutils builds, Python 3.14.9 / 3.15.1 and setuptools in nested host environments. backports.strenum was dropped from the cuda_core lockfiles; the committed locks were already stale after the Python 3.10 drop.
pixi lock --check passes for cuda_pathfinder, cuda_bindings_13 and benchmarks/cuda_bindings. cuda_core and benchmarks/cuda_core still fail it as before this change (consistent with pixi#7215, still open); the root lockfile was not checked.
Not covered: the 12.9.x backport branch has not been checked for the same pins.

Signed-off-by: Ralf Juengling rjuengling@nvidia.com

…ident Raise packages below the fixed versions from the Python security advisory (https://discuss.python.org/t/109576): - ci/pixi-version.env: pixi v0.73.0 -> v0.82.1 - pypa/cibuildwheel v4.3.1 -> v4.3.4 - pypa/gh-action-pypi-publish v1.14.2 -> v1.14.3 - pixi lockfiles: certifi 2026.10.19, pip 26.2.2, uv 0.13.5 and ca-certificates 2026.10.19, updated per package with pixi 0.82.1 (no general refresh, per the lockfile maintenance pause) Known gaps: the source-package host environments in the two benchmarks workspaces still lock uv 0.12.23; they cannot currently be re-solved. Refs NVIDIA#3195 Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>

copy-pr-bot · 2026-10-22T23:38:41Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Follow-up to the eMudhra/emSign CA remediation: lock uv 0.13.5 (and ca-certificates 2026.10.19) in the host environments of the source packages in benchmarks/cuda_bindings and benchmarks/cuda_core. These could not be re-solved with the committed manifests because the benchmark workspaces pin python = "3.14.*" but define no build-variants, so the re-evaluated source packages required python_abi 3.15. The lockfiles were updated with a temporary [workspace.build-variants] python = ["3.14.*"] entry (plus a temporary uv dependency) in each pixi.toml, both reverted afterwards. The manifests are unchanged. Re-solving also moved a few unrelated packages (cuda-pathfinder 1.8.4, CUDA 13.4.92 build 1, python 3.14.9, setuptools, binutils builds). benchmarks/cuda_core/pixi.lock still fails `pixi lock --check`, as it did before this change. Refs NVIDIA#3195 Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>

Andy-Jost · 2026-10-22T23:59:37Z

Ralf, #3197 looks good to me. All seven pin SHAs match the v4.3.4 and v1.14.3 tags. Every certifi, pip, uv and ca-certificates entry in the six lockfiles is at the fixed version.

One gap: CI still runs pip older than 26.2.2. The setup-python builds for 3.10, 3.11 and 3.12 date from Oct 1 and ship the old ensurepip pip. Our floors do not force an upgrade: pip>=25.1 in test-wheel-linux.yml and test-wheel-windows.yml, and pip>=25.3 in coverage.yml and test-sdist-*.yml. Some jobs also run pip right after setup-python with no upgrade:

build-wheel.yml: pip install twine and the pathfinder pip wheel
build-docs.yml, ci-nightly.yml and pre-commit.yml

The pathfinder pip wheel matters most, because it fetches the build backend from PyPI for a wheel we ship. The advisory's fix is a hash-pinned bootstrap after each setup-python step:

python -m pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"

juenglin · 2026-10-22T23:59:43Z

/ok to test a417bd8

Andy-Jost

LGTM - the commented issues could go into this PR or a follow up

coderabbitai · 2026-10-23T00:05:13Z

📝 Summary

Summary by CodeRabbit

Chores
- Updated tools used to build and publish package releases.
- Updated the Pixi environment version and selected tools and certificate data in the lockfiles. Documentation clarifies that these were targeted updates rather than a general lockfile refresh.
- Standardized pip setup across package build, test, coverage, and link-check workflows.
- Updated wheel-building and package-publishing workflow tools to newer versions.

suggestion: The pull request adds pip hardening to CI workflows, updates wheel-build and publishing action versions, and changes the Pixi version pin and maintenance documentation.

Changes: CI toolchain maintenance

Layer / File(s)	Summary
Shared pip hardening and CI integration `.github/actions/harden-pip/action.yml`, `.github/actions/lychee-links/action.yml`, `.github/workflows/ci-nightly.yml`, `.github/workflows/coverage.yml`, `.github/workflows/pre-commit.yml`, `.github/workflows/test-sdist-`, `.github/workflows/test-wheel-`, `.github/actionlint.yaml`	A composite action installs pip 26.2.2 from a wheel URL with a SHA-256 fragment. CI workflows invoke the action. Some workflows pass a virtual-environment Python executable, and several no longer install or upgrade pip separately. The actionlint configuration ignores the missing-ref format diagnostic for self-repository action references in the wheel-test workflows.
Wheel-build workflow updates `.github/workflows/build-wheel.yml`	Three wheel-build steps now use `pypa/cibuildwheel` v4.3.4 instead of v4.3.1. The workflow also invokes pip hardening after Python setup, including conditionally before tests.
Package-publishing action updates `.github/workflows/release-cuda-pathfinder.yml`, `.github/workflows/release.yml`	TestPyPI and PyPI publishing steps in both workflows now use `pypa/gh-action-pypi-publish` v1.14.3 instead of v1.14.2.
Pixi pin and maintenance documentation `ci/pixi-version.env`, `ci/README.md`	The Pixi pin changes from v0.73.0 to v0.82.1. The documentation lists individual updates to `certifi`, `pip`, `uv`, and `ca-certificates` for the eMudhra/emSign CA incident, and states that no general lockfile refresh was performed.

Possibly related issues

suggestion: NVIDIA/cuda-python#3195 describes the eMudhra/emSign incident. This pull request updates related CI pins and documents selected package updates; the issue summary notes that release-branch backports and remaining CI coverage are not established as complete.
suggestion: NVIDIA/cuda-python#2850 requests a mandatory Pixi lockfile freshness check. The changes document that no general lockfile refresh was performed, but do not add the check.
suggestion: NVIDIA/cuda-python#3093 requests dependency and pending-PR checks for cuda-core. The provided summaries do not establish that this pull request completes that checklist.
suggestion: NVIDIA/cuda-python#3094 requests dependency and pending-PR checks for cuda-bindings releases. The provided summaries do not establish that this pull request completes that checklist.

Suggested reviewers: jpascucci-nv

Priority: ➖ Normal

Change: Other

Merge Risk: 🔵 Low · up to 7d93e

Release verification should apply the intended pip security hardening before downloading packages; both gaps are localized fixes.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: afa6249
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: befe61a4-aaa8-4b68-b31a-7e18bcc4719a

📥 Commits

Reviewing files that changed from the base of the PR and between afa6249 and a417bd8.

⛔ Files ignored due to path filters: 6

benchmarks/cuda_bindings/pixi.lock is excluded by !**/*.lock
benchmarks/cuda_core/pixi.lock is excluded by !**/*.lock
cuda_bindings_13/pixi.lock is excluded by !**/*.lock
cuda_core/pixi.lock is excluded by !**/*.lock
cuda_pathfinder/pixi.lock is excluded by !**/*.lock
pixi.lock is excluded by !**/*.lock

📒 Files selected for processing: 5

.github/workflows/build-wheel.yml
.github/workflows/release-cuda-pathfinder.yml
.github/workflows/release.yml
ci/README.md
ci/pixi-version.env

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Andy-Jost noted on NVIDIA#3197 that setup-python ships the old ensurepip pip, and our `pip>=25.1`/`pip>=25.3` floors neither force 26.2.2 nor protect the upgrade download itself. Any pip before 26.2.2 loads its vendored certifi, which trusts the compromised eMudhra/emSign intermediate CAs. Add a reusable `.github/actions/harden-pip` composite action that installs pip 26.2.2 from a sha256-pinned wheel (integrity is guaranteed by the hash even when the bootstrapping pip's trust store is compromised), and invoke it right after each setup-python / `python -m venv` and before the first pip network call in the affected workflows: - build-wheel.yml (both setup-python steps; covers twine and the shipped pathfinder/cuda-python `pip wheel` builds) - test-wheel-linux.yml, test-wheel-windows.yml - test-sdist-linux.yml, test-sdist-windows.yml (drop the now-redundant "pip>=25.3" token) - coverage.yml (all four venv/setup-python jobs; replaces the "pip>=25.3" upgrades) - pre-commit.yml (drops the unprotected `--upgrade pip`) - ci-nightly.yml (CI-tools job on the runner's system python) - lychee-links composite action (covers lychee.yml and build-docs.yml link checks) Refs NVIDIA#3195, NVIDIA#3197 Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>

juenglin · 2026-10-23T00:14:25Z

/ok to test 72dff4d

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: afa6249
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: d08cfa2f-cea1-45c7-b687-1da2a1e36f79

📥 Commits

Reviewing files that changed from the base of the PR and between a417bd8 and 72dff4d.

📒 Files selected for processing: 10

.github/actions/harden-pip/action.yml
.github/actions/lychee-links/action.yml
.github/workflows/build-wheel.yml
.github/workflows/ci-nightly.yml
.github/workflows/coverage.yml
.github/workflows/pre-commit.yml
.github/workflows/test-sdist-linux.yml
.github/workflows/test-sdist-windows.yml
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

github-actions · 2026-10-23T00:32:28Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-3197/
https://nvidia.github.io/cuda-python/pr-preview/pr-3197/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-3197/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-3197/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

CodeRabbit noted on NVIDIA#3197 that the wheel test jobs check out inputs.sha, which in nightly runs is the head of the last successful main CI run and can predate the harden-pip action. A ./.github/actions reference resolves against that checkout, so every nightly wheel-test job would fail at "Harden pip" until a main CI run that includes the action succeeds (the same pitfall NVIDIA#3052 worked around for download-artifact-retry). Reference the action with GitHub's self-repository syntax instead, which resolves at the running workflow's commit regardless of the checkout. It requires runner 2.336.0 or newer; the GPU runners report 2.337.0. actionlint v1.7.12, the latest release, does not yet recognize $/ (rhysd/actionlint#711), so add narrow per-file ignores for it. Refs NVIDIA#3195, NVIDIA#3197 Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>

juenglin · 2026-10-23T01:10:10Z

/ok to test 7d93e3b

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (2)

🟡 Minor · important: Harden the Pathfinder verification environments… · release-cuda-pathfinder.yml:252-260

.github/workflows/release-cuda-pathfinder.yml:252-260
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

important: Harden the Pathfinder verification environments before installing packages.

Both verification jobs create a virtual environment and immediately run pip install. Neither job invokes harden-pip, so the release checks use the venv-bootstrap pip instead of pinned pip 26.2.2. The checks can therefore pass with the outdated pip that this change is intended to replace.
Suggested fix
-      - name: Install from TestPyPI and verify
+      - name: Create verification environment
         run: |
           python3 -m venv /tmp/verify
+      - name: Harden pip
+        uses: $/.github/actions/harden-pip
+        with:
+          python: /tmp/verify/bin/python
+      - name: Install from TestPyPI and verify
+        run: |
           source /tmp/verify/bin/activate
           max_attempts=6
           retry_seconds=30
Apply the same change to verify-pypi, using /tmp/verify/bin/python before its first pip install.

🟡 Minor · 🎯 Functional Correctness · release.yml:330-333

.github/workflows/release.yml:330-333
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

important: Harden the verification environments before installing from TestPyPI or PyPI.

The release verification jobs create a virtual environment and run pip install without invoking harden-pip. Those installs therefore use the version bootstrapped by venv, not the hash-pinned pip 26.2.2 used by the other updated workflows. Pass each verification environment's Python executable to harden-pip before the first install. Otherwise, release verification does not use the pinned pip version for its package downloads.

ℹ️ Review info

⚙️ Run configuration

Source mode: Auto
Repository: NVIDIA/cuda-python
Branch: main
File: .coderabbit.yaml
Commit: afa6249
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: ec408da4-3dc9-467f-b970-aeaa4b22dfac

📥 Commits

Reviewing files that changed from the base of the PR and between 72dff4d and 7d93e3b.

📒 Files selected for processing: 3

.github/actionlint.yaml
.github/workflows/test-wheel-linux.yml
.github/workflows/test-wheel-windows.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

github-actions Bot added CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module cuda.core Everything related to the cuda.core module cuda.pathfinder Everything related to the cuda.pathfinder module labels Oct 22, 2026

juenglin requested review from Andy-Jost and rparolin October 22, 2026 23:43

juenglin marked this pull request as ready for review October 22, 2026 23:59

juenglin self-assigned this Oct 23, 2026

juenglin added this to the cuda.bindings next milestone Oct 23, 2026

Andy-Jost approved these changes Oct 23, 2026

View reviewed changes

coderabbitai Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread .github/workflows/build-wheel.yml

rwgk approved these changes Oct 23, 2026

View reviewed changes

Comment thread ci/pixi-version.env

coderabbitai Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread .github/workflows/test-wheel-linux.yml Outdated

juenglin enabled auto-merge (squash) October 23, 2026 01:06

coderabbitai Bot reviewed Oct 23, 2026

View reviewed changes

Conversation

juenglin commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commits

Notes for reviewers

Uh oh!

copy-pr-bot Bot commented Oct 22, 2026

Uh oh!

Andy-Jost commented Oct 22, 2026

Uh oh!

juenglin commented Oct 22, 2026

Uh oh!

Andy-Jost left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

juenglin commented Oct 23, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 23, 2026

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

juenglin commented Oct 23, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

juenglin commented Oct 22, 2026 •

edited

Loading

Andy-Jost left a comment •

edited

Loading

coderabbitai Bot commented Oct 23, 2026 •

edited

Loading