apache / apache/airflow

should we normalize scheme and hostname of URI to be lower?

Open
#25,186 26 comments 0 reactions 0 assignees View on GitHub
airflow3.x candidate area:data-aware-scheduling priority:medium
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

datasets are defined by their URI

we don't really spelly out what exactly our URI standard is, apart from reserving the `airflow` scheme and ascii

for websites, the scheme and hostname are case insensitive, while everything else is not.

should we normalize scheme and hostname or allow case sensitive differentiation?

my inclination is, we should allow everything to be case sensitive, except _possibly_ the scheme. and the reason is, we don't know exactly what "hostname" will mean for a dataset. if, for example, it's a database object, it could be case sensitive. scheme.

if we do implement some normalization, we introduce somewhat of a problem when we do sqlalchemyf lookups by URI because we can't just do `Dataset.uri == uri`; we'd have to normalize the incoming URI value first. One possibility is to split out the scheme into a different column in the db that has CI collation, which would avoid this issue, though it would force you to decompose. but this is messy too.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Dataset URI handling and the SQLAlchemy lookup described in the issue. No file or test is named; first locate the current URI equality path and determine how normalization would affect database lookups. Done means the URI standard and case-sensitivity policy are decided, with the resulting behavior covered in the relevant implementation and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sqlalchemy
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.