OpenEnergyPlatform / OpenEnergyPlatform/open-MaStR
Dates with a year before 1000 are zero-padded on import
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 141
- Forks
- 35
- Avg merge
- 5d 5h
- Merged PRs (30d)
- 5
Description
(Not required for v1.0.0)
Description of the issue
cast_date_columns_to_string in open_mastr/xml_download/utils_write_to_database.py:317 pads the year of Date columns before writing them to SQLite:
df.loc[mask, column.name] = df.loc[mask, column.name].str.zfill(10)
This was added in #728 to fix #719. It makes the stored string readable again, but it also decides what the year is: 205-12-06 becomes 0205-12-06 and 24-01-04 becomes 0024-01-04. Both source values are typos in the MaStR, where 205 is most likely 2005 or 2025 and 24 is most likely 2024. The padded value is a date that does not appear in the register, it is indistinguishable from a correct one, and nothing marks it as uncertain. open-mastr mirrors the MaStR and does not reinterpret it, so it should not invent a year here.
DateTime columns are not padded, so the same typo is handled differently depending on the column type:
| source value | stored in a Date column |
stored in a DateTime column |
|---|---|---|
205-12-06 |
0205-12-06, exported as the year 205 |
205-12-06 00:00:00.000000 |
24-01-04 |
0024-01-04, exported as the year 24 |
24-01-04 00:00:00.000000 |
A full download of 2026-09-15 holds 91 values whose year is not four digits, spread over 40,042,729 rows:
| table | column | type | values |
|---|---|---|---|
Marktakteure |
Taetigkeitsbeginn |
DATE |
58 |
EinheitenSolar |
DatumDesBetreiberwechsels |
DATE |
25 |
EinheitenStromSpeicher |
DatumDesBetreiberwechsels |
DATE |
5 |
EinheitenVerbrennung |
DatumDesBetreiberwechsels |
DATE |
2 |
EinheitenSolar |
InbetriebnahmedatumAmAktuellenStandort |
DATETIME |
1 |
So 90 of the 91 values are stored with an invented year, and only the single DateTime value is reported and exported as empty by Mastr.to_csv by #801.
Current behavior: a Date whose year has fewer than four digits is silently changed into a year that the MaStR does not contain, while the same value in a DateTime column is kept as it is and exported as empty.
Expected behavior: both column types are treated the same way, and no year is invented. Since Mastr.to_csv already reports unreadable dates and exports them as empty cells, storing the value unchanged would make the two types consistent without losing information that the register actually provides.
This was found while working on #800 , a solution was temporarily included in 767f6746e0dae8919bbc5edbd221f11c21d51495 but reverted.
Steps to Reproduce
- Check out
develop. - Run the snippet below, which calls the import helper directly and needs no download.
import numpy as np
import pandas as pd
from sqlalchemy import Column, Date, DateTime, MetaData, String, Table
from open_mastr.xml_download.utils_write_to_database import cast_date_columns_to_string
for column_type in (Date, DateTime):
table = Table(
"einheitensolar",
MetaData(),
Column("EinheitMastrNummer", String, primary_key=True),
Column("DatumDesBetreiberwechsels", column_type),
)
df = pd.DataFrame(
{
"EinheitMastrNummer": ["1"],
# the MaStR contains 205-12-06, meaning 2005 or 2025
"DatumDesBetreiberwechsels": np.array(
["0205-12-06"], dtype="datetime64[us]"
),
}
)
print(column_type.__name__, cast_date_columns_to_string(table, df).iloc[0, 1])
- The
Datecolumn prints0205-12-06, theDateTimecolumn prints205-12-06 00:00:00.000000.
Alternatively, run a full Mastr().download() and look for the padded years in the resulting database. A four-digit year in the MaStR never starts with a zero, so every match was padded on import:
SELECT Taetigkeitsbeginn, count(*)
FROM Marktakteure
WHERE Taetigkeitsbeginn GLOB '0[0-9][0-9][0-9]-*'
GROUP BY Taetigkeitsbeginn;
Ideas of solution
Remove the zero-padding and store the value as it is. Mastr.to_csv already detects dates that cannot be read back, warns about them and exports the cell as empty (#801), so Date and DateTime columns would behave identically and no invented year would enter the database. Disadvantage: 90 values that currently carry a wrong but parseable date would be exported as empty cells instead. Advantage: the database no longer contains dates that the register does not contain, and a user who needs the raw value can still read it from the database, where it is stored verbatim.
Context and Environment
- Version used:
develop, HEAD: 1c901d1354589acbb9ee372bcc4853630be72df9 - Operating system: Ubuntu 26.04 LTS
- Environment setup and (python) version: conda, Python 3.14.6
Workflow checklist
- I have checked the documentation and confirmed this issue is not already addressed there.
- I am aware of the workflow in CONTRIBUTING.md
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in open_mastr/xml_download/utils_write_to_database.py:317 at cast_date_columns_to_string and run the supplied pandas/SQLAlchemy reproduction for Date and DateTime columns. Verify that the import path preserves source values without inventing a four-digit year and remains consistent with Mastr.to_csv's handling of unreadable dates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python, sqlalchemy, sqlite
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100