Skip to content

fix: write timezone-aware coordinates in Model.to_netcdf - #1021

Merged
FabianHofmann merged 1 commit into
PyPSA:masterfrom
MaykThewessen:fix/netcdf-tz-aware-coords
Oct 30, 2026
Merged

FabianHofmann merged 1 commit into
PyPSA:masterfrom
MaykThewessen:fix/netcdf-tz-aware-coords

Conversation

@MaykThewessen

@MaykThewessen MaykThewessen commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Note

AI-assisted. Claude wrote the code and tests, and ran the checks listed below. I reviewed the diff.

Changes proposed in this Pull Request

Follow-up to #897. Since that fix a model with timezone-aware coords builds and solves, but Model.to_netcdf() then raises TypeError: Cannot interpret 'datetime64[us, Europe/Amsterdam]' as a data type, because netCDF has no timezone-aware datetime type. This also breaks solve(remote=...), which goes through to_netcdf. PyPSA runs into it as soon as it allows tz-aware snapshots (PyPSA/PyPSA#983).

to_netcdf now writes tz-aware datetimes as naive UTC and stores the zone names in a _linopy_tz attribute, and read_netcdf restores them. This covers dimension indexes, MultiIndex levels, auxiliary coordinates (the flat-dimension pattern of v1), CSR-backed constraints including their aux coords, and tz-aware data variables in Model.parameters. Files without the attribute read as before.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.
What was checked (AI-generated)
  • New test_model_to_netcdf_tz_aware_coords in test/test_io.py, for UTC and Europe/Amsterdam over the 2020-03-29 DST change: flat index, frozen (CSR) constraint and MultiIndex level, plus a tz-aware data variable in Model.parameters. It asserts assert_model_equal and that the timezone comes back. All 10 cases fail on master (the 2 MultiIndex cases under v1 are skipped, like the existing MultiIndex fixture).
  • New test_model_to_netcdf_tz_aware_aux_coords (dense and frozen, 8 cases): all fail on master with the TypeError above. Dense models already drop aux coords on read, also with naive datetimes; that is unchanged here.
  • Full test suite, branch vs master: 5729 vs 5709 passed: the 18 new cases, plus 2 test_highs_keyboard_interrupt tests that failed on the master run only (flaky). The branch has no failures that master lacks; the rest come from the environment (missing oetc extras, HiGHS without SOS).
  • End to end with a PyPSA network on tz-aware snapshots: n.model.to_netcdf() then read_netcdf() gives back equal snapshot indexes with the zone and an equal solution.
  • Not in this PR: freezing a constraint with a MultiIndex dimension already fails on master (Grid.from_dataset, TypeError: Must pass list-like as names), so the test leaves that combination out.

🤖 Generated with Claude Code

@codspeed

codspeed Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing MaykThewessen:fix/netcdf-tz-aware-coords (6a861d3) with master (8189160)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@FabianHofmann

Copy link
Copy Markdown
Collaborator

Thanks @MaykThewessen the feature is useful and we need it for PyPSA/PyPSA#983. But I'd like a simpler approach before merging. The code looks hacky and does not draw the right boundary


iterated with Claude on this, here is the ouptut

Right now the timezone handling lives in three places (common.py for indexes and MultiIndex levels, csr.py for aux coords, io.py for the rest), and each one stores the zone in a different attribute format. That is hard to maintain, and we're reworking the CSR backend right now.

to_netcdf already merges everything into one dataset (MultiIndexes are flattened by with_prefix), and read_netcdf loads it with a single xr.load_dataset. So I think two small helpers at that boundary are enough:

  • before writing: for every variable in the merged dataset that is tz-aware, convert to naive UTC and set _linopy_tz = "<zone>" on that variable
  • after loading: for every variable with _linopy_tz, localize back

That covers indexes, MultiIndex levels, CSR _index/aux vars and parameters in one go, and leaves common.py and csr.py untouched. Please also drop the code comments and getattr, and skip special MultiIndex handling, since MultiIndex dimensions are removed in 1.0.

@MaykThewessen

MaykThewessen commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Note: AI-assisted.

Thanks @FabianHofmann, agreed. I rewrote it along those lines in d38fd4f: two helpers in io.py, _tz_to_naive_utc right before ds.to_netcdf and _tz_from_naive_utc right after xr.load_dataset, with the zone in _linopy_tz on each variable. common.py and csr.py are untouched. There is no getattr, no MultiIndex handling and no comments. Tests cover dimension and aux coords, frozen constraints and Model.parameters, for UTC and Europe/Amsterdam across the DST switch.

One detail: on read, a dimension coordinate comes back as a tz-aware DatetimeIndex, everything else as an object array of tz-aware timestamps. The CSR reader builds its indexes from .values, which drops the zone for a tz-aware dtype.

Checked by machine: the diff against upstream/master (only io.py, test_io.py and the release notes change) and CI on d38fd4f (all checks green).

netCDF has no timezone-aware datetime type, so to_netcdf raised on
tz-aware indexes, auxiliary coordinates and parameters. Handle it at the
netcdf boundary only: _tz_to_naive_utc writes naive UTC with the zone in a
_linopy_tz attribute right before ds.to_netcdf, and _tz_from_naive_utc
restores it in read_netcdf, auxiliary coordinates included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaykThewessen
MaykThewessen force-pushed the fix/netcdf-tz-aware-coords branch from 2636e72 to 6a861d3 Compare October 28, 2026 12:36
@FabianHofmann

Copy link
Copy Markdown
Collaborator

thanks @MaykThewessen

@FabianHofmann
FabianHofmann merged commit 51166de into PyPSA:master Oct 30, 2026
22 checks passed
FabianHofmann added a commit that referenced this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants