Repository navigation
Optimize matrix testing in CI by using hatch testing instead of a full matrix of runners - #517
Merged
Merged
Conversation
Lately, MacOS runners are oversolicited and jobs frequently time out. Use hatch test -a to run all tests in one MacOS job. Squashes all the following commits: - ci: use hatch to run matrix testing - ci: set SETUPTOOLS_SCM_PRETEND_VERSION before running hatch - ci: cache the uv cache in gh-actions - ci: let astral-sh/setup-uv handle caching the uv cache - ci: use uvx to run hatch instead of having a separate install
When testing using `hatch test -a`: - test_studio: take a 1% sample instead of a 10% sample - test_convert: take a 10% sample instead of testing all When testing just with pytest, reduce test_studio to a 5% sample but keep test_convert as it was before.
Contributor
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Goal?
Lately, MacOS runners are hard to get a hold of, and our matrix tests constantly fail because some of the MacOS jobs time out before they get picked up.
Solution: use
hatch test -ato cover the different versions of PythonSo:
Also, hatch testing hightlights that some of our redundant tests end up being slow, with test_studio, test_langs and test_cli all running over all data in
tests/public/data. When running with hatch, let's down sample some more to keep test times more reasonable.Fixes?
Recent flakiness of our matrix testing
Feedback sought?
sanity checking
Priority?
medium-high
Tests added?
nothing but!
How to test?
Locally:
uvx hatch test -aor
uvx hatch test -a -- --durations=10to see which tests are slow.In CI look at the matrix test workflows I ran.
Confidence?
high
Version change?
no