apache / apache/airflow

StartOfDayMapper does not work on assets that return a Date

Open
#70,762 1 comment 0 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Task SDK

### Apache Airflow version

3.3.0

### What happened and how to reproduce it?

** Issue Description **
StartOfDayMapper defaults input_format to "%Y-%m-%dT%H:%M:%S". If an upstream asset emits a partition key that's already at day granularity (e.g. "2026-07-30"), calling to_downstream() without explicitly overriding input_format raises an error.

As a result, the mapper must be explicitly configured with the correct input_format up front. This prevents flexible handling of a mix of partition keys at different granularities (e.g. some upstream assets emitting day-level keys, others hour-level) without requiring separate, explicitly-configured mapper instances for each.

** Repro **
from airflow.partition_mappers.temporal import StartOfDayMapper

# --- Case 1: full timestamp input (matches the default input_format) ---
mapper = StartOfDayMapper() # default input_format="%Y-%m-%dT%H:%M:%S"
print(mapper.to_downstream("2026-07-30T14:22:05")) # → "2026-07-30"

# --- Case 2: date-only input (works) ---
mapper_date_only = StartOfDayMapper(input_format="%Y-%m-%d")
print(mapper_date_only.to_downstream("2026-07-30")) # → "2026-07-30"

# --- Case 3: date-only input WITHOUT telling it the format (reproduces the error) ---
mapper_default = StartOfDayMapper() # still expects full timestamp
print(
mapper_default.to_downstream("2026-07-30")
) # → ValueError: time data '2026-07-30' does not match format '%Y-%m-%dT%H:%M:%S'

### What you think should happen instead?

Either:
Partition keys already at day granularity should be passed through unchanged, similar to IdentityMapper,

or

StartOfDayMapper should output keys in its own default input_format (i.e. "%Y-%m-%dT%H:%M:%S", with the time component zeroed to T00:00:00), so that its output can be fed back into another instance of the same mapper (chaining).

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with StartOfDayMapper in airflow.partition_mappers.temporal and run the three reproduction cases from the issue. First confirm whether date-only keys should pass through or be normalized to a zeroed timestamp, then preserve the existing full-timestamp behavior and add regression coverage for the selected contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.