Skip to content

Don't skip non-ASCII files in check mode on Windows - #2740

Merged
DanielNoord merged 1 commit into
PyCQA:mainfrom
nk-batman21:fix-check-devnull-encoding
Oct 25, 2026
Merged

DanielNoord merged 1 commit into
PyCQA:mainfrom
nk-batman21:fix-check-devnull-encoding

Conversation

@nk-batman21

Copy link
Copy Markdown
Contributor

On Windows with Python 3.11 to 3.14 (without UTF-8 mode), isort --check-only lets an unsorted file pass if it contains a character the ANSI code page cannot encode, such as an emoji, CJK text or ✓. For a file unsorted.py containing

import sys
import os

print('✓ done')

running python -m isort --check-only unsorted.py on a cp1252 machine prints UserWarning: Unable to parse file unsorted.py due to 'charmap' codec can't encode character '\u2713' in position 32: character maps to <undefined> and exits 0. isort.check_file() raises UnicodeEncodeError on the same file. 8.0.1 reports the file as incorrectly sorted and exits 1. The same thing happens in this repo: on Windows, isort --profile hug --check isort/ tests/ skips seven of the test files with this warning.

This is a regression from 9.0.0: since fed6c0c (Remove _EmptyIO, #2601), check_stream() writes the sorted output to open(os.devnull, "w"), which uses the locale encoding. That output is thrown away, so this change opens it as UTF-8.

The new test in tests/unit/test_api.py fails without the change on Windows with a cp1252 locale (checked on 3.11 and 3.14). On Linux and macOS it passes either way because the locale encoding is already UTF-8.

Ran: pytest tests/unit/test_api.py (18 passed) and coverage run -m pytest tests/unit/ -s (713 passed, 5 skipped, coverage 99%) on Windows with Python 3.14.
Checked: mypy, flake8, ruff check, ruff format --check, bandit -r isort/ and isort --profile hug --check on the changed files.

`check_stream()` sorts into `open(os.devnull, "w")`, which uses the locale
encoding. On Windows with Python 3.11 to 3.14 that is usually an ANSI code
page such as cp1252, so a file containing a character it cannot encode (an
emoji, CJK text, a check mark) raises `UnicodeEncodeError`. `isort --check-only`
turns that into an "Unable to parse file" warning and exits 0 even when the
imports are unsorted, and `isort.check_file()` raises instead of returning
False. Before `_EmptyIO` was removed the output went to an in-memory sink, so
8.0.1 reported these files correctly.

The output is discarded, so open devnull as UTF-8.
@codecov

codecov Bot commented Oct 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.38%. Comparing base (a514c71) to head (2e14bed).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2740   +/-   ##
=======================================
  Coverage   99.38%   99.38%           
=======================================
  Files          40       40           
  Lines        3274     3274           
  Branches      706      706           
=======================================
  Hits         3254     3254           
  Misses         12       12           
  Partials        8        8           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The narrow encoding fix addresses the reported failure without changing sorting behavior, and the regression test covers both check outcomes.

No open issues 🎉

What changed in this PR

Fixes a Windows check-mode regression by explicitly using UTF-8 for discarded sorting output, preventing non-ASCII files from being skipped because of locale encoding errors.

Changes:

  • Opens the null output stream with UTF-8 encoding.
  • Adds regression coverage for sorted and unsorted files containing a non-ASCII character.
File Description
tests/​unit/​test_api.py Tests both check results with non-ASCII content.
isort/​api.py Uses UTF-8 for discarded check-mode output.

Review effort: Balanced

@DanielNoord
DanielNoord added this pull request to the merge queue Oct 25, 2026
Merged via the queue into PyCQA:main with commit c03ad5d Oct 25, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants