dlt-hub / dlt-hub/dlt

improve destination fingerprinting and info

Open
#751 0 comments 0 reactions 0 assignees View on GitHub
tech-debt
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 15h
Merged PRs (30d)
38

Description

**Background**
Building on #746 we want a robust fingerprinting and destination info to include in `LoadInfo` and pipeline traces. The fingerprinting is used to anonymously identify cloud destinations and allows to build proper tracing and data lineage

**Tasks**
1. * [x] improve `filesystem` fingerprint by including only the schema and netloc. currently we also include path in the bucket which is too much. for local filesystem configuration return a hash of empty string.
2. * [ ] add method to destination factory that will return a destination info as named tuple/dict
3. * [ ] improve fingerprinting of destinations that run locally (duckdb/filesystem/postgres)

**Implementation**
Please do 3 PRs for 3 tasks

destination info contains:
- destination type, name, environment and fingerprint
- destination str() representation (which is display safe)
- local/remote flag

local destination:
for destinations that may run locally (duckdb, postgres, waeviate, quadrant, filesystem etc.) we should start generating fingerprints
- if destination runs locally (always duckdb, `file://` on filesystem, localhost and 127.0.0.1 + ip6 localhost if connection string) a local flag must be set to true in info
- fingerprint must include `.anonymous_id` as used by telemetry and path to file/database when applicable
- `.anonymous_id` should be detached from telemetry code and become independent (maybe part of `paths.py` module)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.