[ACTION REQUIRED from contributors] Reducing our GitHub Actions usage: please rebase your open PRs

[yihua]

Hi all,

ASF Infra has disabled GitHub Actions on apache/hudi temporarily due to a surge in GitHub Actions runner usage. Nothing can be verified or merged until that comes down.

PR #19514 reduces GitHub Actions usage by keeping only the necessary Java CI jobs. This is an interim measure to unblock PR merging.

For contributors, two things to act on:

1. Rebase your open PRs onto the latest master after PR chore(ci): trim the CI matrix and reset required checks to unblock PRs #19514 is merged. Checks from the old matrix do not carry over, by design, so a PR will not be mergeable until it has a fresh run.
2. Be deliberate about git pushes. Every push starts a full GitHub Actions run. Please squash and self-review locally, run the relevant module tests before pushing, and avoid re-pushing or re-running jobs just to retrigger CI.

Meanwhile, the Java CI in GitHub Actions is being enhanced to provide stronger test coverage, including Spark core tests (#19082), and a more efficient way of running tests on different Spark releases. Follow-ups on a more durable setup are welcome on this thread.

Thanks,

• Ethan

[wombatu-kun]

A few measurements, so the follow-ups here can be argued against numbers rather than intuition. Method: sum completed_at - started_at over non-skipped jobs, since billable comes back empty for public repos.

A full Java CI run is 69 jobs, 67 of them concurrent, and 2,610 runner-minutes (43.5 h). At 255 runs in the week of Jul 27 that is ~7,200 runner-hours against the 4,200 h weekly limit, and the Spark datasource jobs are 78% of it. Everything outside bot.yml is rounding error.

Two things that outlive the current trim:

• Concurrency is a separate limit, MUST 20 and SHOULD 15, and nothing tracks it today. chore(ci): trim the CI matrix and reset required checks to unblock PRs #19514 lands at 17 job instances, already above the SHOULD, and test: Spark core tests #19082 adds a 7-entry matrix on top of the surviving jobs, which puts us back over the MUST. Worth pinning with an explicit strategy.max-parallel rather than leaving it to the job count.
• Nothing stops the matrix from regrowing. It went from 55 jobs and 1,845 min on Jun 4 to 69 jobs and 2,610 min on Aug 4, +41% in two months, almost entirely from added Spark versions; we carry 6 Spark and 5 Flink profiles today. test: Spark core tests #19082 tiers the versions we have now, but nothing in it says what happens when the next minor lands. Should the tiering carry a stated cap, say full suites on one 3.x and one 4.x only, so a new Spark minor adds a core lane instead of a full one?

On the rebase ask I left a comment on #19514 instead, since it turns on that diff.

[vinothchandar]

Ultimately, we need to reduce the heaviness of the CI itself. I am not sure if these announcements are heeded by everyone and actioned on in a timely fashion.

We need to start looking into cutting down some combinations (scala, spark versions, flink versions and such).