OpenEnergyPlatform / OpenEnergyPlatform/open-MaStR

Dates with a year before 1000 are zero-padded on import

Open
#802 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

:bug: bug
Dominant language
Python
Stars
141
Forks
35
Avg merge
5d 5h
Merged PRs (30d)
5

Description

(Not required for v1.0.0)

Description of the issue

cast_date_columns_to_string in open_mastr/xml_download/utils_write_to_database.py:317 pads the year of Date columns before writing them to SQLite:

df.loc[mask, column.name] = df.loc[mask, column.name].str.zfill(10)

This was added in #728 to fix #719. It makes the stored string readable again, but it also decides what the year is: 205-12-06 becomes 0205-12-06 and 24-01-04 becomes 0024-01-04. Both source values are typos in the MaStR, where 205 is most likely 2005 or 2025 and 24 is most likely 2024. The padded value is a date that does not appear in the register, it is indistinguishable from a correct one, and nothing marks it as uncertain. open-mastr mirrors the MaStR and does not reinterpret it, so it should not invent a year here.

DateTime columns are not padded, so the same typo is handled differently depending on the column type:

source value stored in a Date column stored in a DateTime column
205-12-06 0205-12-06, exported as the year 205 205-12-06 00:00:00.000000
24-01-04 0024-01-04, exported as the year 24 24-01-04 00:00:00.000000

A full download of 2026-09-15 holds 91 values whose year is not four digits, spread over 40,042,729 rows:

table column type values
Marktakteure Taetigkeitsbeginn DATE 58
EinheitenSolar DatumDesBetreiberwechsels DATE 25
EinheitenStromSpeicher DatumDesBetreiberwechsels DATE 5
EinheitenVerbrennung DatumDesBetreiberwechsels DATE 2
EinheitenSolar InbetriebnahmedatumAmAktuellenStandort DATETIME 1

So 90 of the 91 values are stored with an invented year, and only the single DateTime value is reported and exported as empty by Mastr.to_csv by #801.

Current behavior: a Date whose year has fewer than four digits is silently changed into a year that the MaStR does not contain, while the same value in a DateTime column is kept as it is and exported as empty.

Expected behavior: both column types are treated the same way, and no year is invented. Since Mastr.to_csv already reports unreadable dates and exports them as empty cells, storing the value unchanged would make the two types consistent without losing information that the register actually provides.

This was found while working on #800 , a solution was temporarily included in 767f6746e0dae8919bbc5edbd221f11c21d51495 but reverted.

Steps to Reproduce

  1. Check out develop.
  2. Run the snippet below, which calls the import helper directly and needs no download.
import numpy as np
import pandas as pd
from sqlalchemy import Column, Date, DateTime, MetaData, String, Table

from open_mastr.xml_download.utils_write_to_database import cast_date_columns_to_string

for column_type in (Date, DateTime):
    table = Table(
        "einheitensolar",
        MetaData(),
        Column("EinheitMastrNummer", String, primary_key=True),
        Column("DatumDesBetreiberwechsels", column_type),
    )
    df = pd.DataFrame(
        {
            "EinheitMastrNummer": ["1"],
            # the MaStR contains 205-12-06, meaning 2005 or 2025
            "DatumDesBetreiberwechsels": np.array(
                ["0205-12-06"], dtype="datetime64[us]"
            ),
        }
    )
    print(column_type.__name__, cast_date_columns_to_string(table, df).iloc[0, 1])
  1. The Date column prints 0205-12-06, the DateTime column prints 205-12-06 00:00:00.000000.

Alternatively, run a full Mastr().download() and look for the padded years in the resulting database. A four-digit year in the MaStR never starts with a zero, so every match was padded on import:

SELECT Taetigkeitsbeginn, count(*)
FROM Marktakteure
WHERE Taetigkeitsbeginn GLOB '0[0-9][0-9][0-9]-*'
GROUP BY Taetigkeitsbeginn;

Ideas of solution

Remove the zero-padding and store the value as it is. Mastr.to_csv already detects dates that cannot be read back, warns about them and exports the cell as empty (#801), so Date and DateTime columns would behave identically and no invented year would enter the database. Disadvantage: 90 values that currently carry a wrong but parseable date would be exported as empty cells instead. Advantage: the database no longer contains dates that the register does not contain, and a user who needs the raw value can still read it from the database, where it is stored verbatim.

Context and Environment

  • Version used: develop, HEAD: 1c901d1354589acbb9ee372bcc4853630be72df9
  • Operating system: Ubuntu 26.04 LTS
  • Environment setup and (python) version: conda, Python 3.14.6

Workflow checklist

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in open_mastr/xml_download/utils_write_to_database.py:317 at cast_date_columns_to_string and run the supplied pandas/SQLAlchemy reproduction for Date and DateTime columns. Verify that the import path preserves source values without inventing a four-digit year and remains consistent with Mastr.to_csv's handling of unreadable dates.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python, sqlalchemy, sqlite
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.