Repository navigation
spec: derive dimensions from data and deduplicate repeated labels #1071
Description
Note
The following content was generated by AI.
Found in the API review for #1068 (stacked on #958); linopy.spec is unreleased.
Describe the feature you'd like to see
Three related gaps in how relations and dimensions (sets) take their data:
- A bare relation, with key columns only and no value column, is refused at build time, though mathspec accepts it. The PyPSA example
Process_maintenance_cover/Generator_maintenance_coverneeds one. - A relation cannot come from a two-column DataFrame, the natural shape of a mapping table; only an indexed Series, dict or DataArray works.
- A dimension is never derived from data: it must be passed explicitly, even when a relation already lists its labels, and a table column with repeated labels is refused instead of deduplicated.
Minimal reproducible example
import warnings
import pandas as pd
import linopy
warnings.simplefilter("ignore", linopy.EvolvingAPIWarning)
def attempt(label, spec, sources):
try:
linopy.Model.from_spec(spec, sources)
print("OK ", label)
except Exception as e:
print("FAIL", label, "->", type(e).__name__, e)
MAPPED = {
"dimensions": {"generator": {}, "bus": {}},
"relations": {"gen_bus": {"key": "generator", "values": "bus"}},
"parameters": {"cap": {"dims": ["generator"]}},
"variables": {"x": {"dims": ["bus"], "bounds": {"lower": 0}}},
"constraints": {"c": {"dims": ["bus"], "expression": "x <= sum(cap, over=generator, by=gen_bus[bus])"}},
"objective": {"sense": "maximize", "expression": "sum(x)"},
}
tbl = pd.DataFrame({"generator": ["g1", "g2", "g3"], "bus": ["n", "s", "n"], "cap": [1.0, 2.0, 3.0]})
cap = tbl.set_index("generator")["cap"]
attempt("relation as indexed Series", MAPPED, {"generator": tbl.generator, "bus": ["n", "s"], "gen_bus": tbl.set_index("generator")["bus"], "cap": cap})
attempt("relation as two-column DataFrame", MAPPED, {"generator": tbl.generator, "bus": ["n", "s"], "gen_bus": tbl[["generator", "bus"]], "cap": cap})
attempt("set 'bus' derived from the relation (no 'bus' key)", MAPPED, {"generator": tbl.generator, "gen_bus": tbl.set_index("generator")["bus"], "cap": cap})
attempt("set 'generator' from a column with duplicates", MAPPED, {"generator": pd.Series(["g1", "g2", "g3", "g1"]), "bus": ["n", "s"], "gen_bus": tbl.set_index("generator")["bus"], "cap": cap})
BARE = """
dimensions:
t: {dtype: int}
relations:
cover: {key: {start: t, covered: t}}
variables:
start: {dims: [t], bounds: {lower: 0, upper: 1}}
on: {dims: [t], bounds: {lower: 0, upper: 1}}
constraints:
c: {dims: [t], expression: "on == sum(start, over=start, by=cover[covered])"}
objective: {sense: minimize, expression: "sum(start)"}
"""
cover = pd.DataFrame({"start": [0, 0, 1, 1, 2], "covered": [0, 1, 1, 2, 2]})
attempt("bare relation (key columns only)", BARE, {"t": [0, 1, 2], "cover": cover})Output on spec-simplification:
Output
OK relation as indexed Series
FAIL relation as two-column DataFrame -> SpecDataError relation 'gen_bus': cannot adapt DataFrame to a map; pass a pandas Series indexed by 'generator', a dict keyed by 'generator' labels, or a 1-D DataArray over 'generator'.
FAIL set 'bus' derived from the relation (no 'bus' key) -> SpecDataError dimension 'bus' has no index: pass its labels under key 'bus' as a pandas Index, a list, a tuple, a set (read sorted), a 1-D numpy or pandas array, a pandas Series or a 1-D DataArray. The index is what says which labels exist, and without one a mistyped label is indistinguishable from a new one.
FAIL set 'generator' from a column with duplicates -> SpecDataError dimension 'generator' lists 'g1' more than once. A dimension's members are a set: each label appears once, in the order the source gives it.
FAIL bare relation (key columns only) -> SpecDataError relation 'cover' keys ['start', 'covered'] into [], and linopy reads a relation over two columns: one key column mapping into one value column. A bare relation, several key columns or several value columns are not supported yet; split it into single-valued relations.
Expected
The bare relation builds. The two-column DataFrame reads like the indexed Series. Whether a dimension may be derived from a relation or a column (and how its order is fixed) is a design decision, but the current refusal should at least be a conscious one.
Proposal
Read relations as row sets (any number of key columns, zero or one value column) from DataFrames, and decide on an opt-in rule for deriving a dimension's labels from data.
Activity
FabianHofmann commentedon Oct 30, 2026
Note
Generated by AI after checking against current master.
Status on master: two-column DataFrame relations and bare relations (
missing: absent) now build. Still refused: deriving a dimension from a relation and deduplicating repeated labels.[-]spec: bare relations, relations from two-column DataFrames, and sets derived from data[/-][+]spec: derive dimensions from data and deduplicate repeated labels[/+]spec: one input shape per kind of data, with polars frames (#1093)
spec: document the holes operators create (#1094)
FabianHofmann commentedon Nov 2, 2026
close as partially fixed through stricter convention