improve destination fingerprinting and info
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
**Background**
Building on #746 we want a robust fingerprinting and destination info to include in `LoadInfo` and pipeline traces. The fingerprinting is used to anonymously identify cloud destinations and allows to build proper tracing and data lineage
**Tasks**
1. * [x] improve `filesystem` fingerprint by including only the schema and netloc. currently we also include path in the bucket which is too much. for local filesystem configuration return a hash of empty string.
2. * [ ] add method to destination factory that will return a destination info as named tuple/dict
3. * [ ] improve fingerprinting of destinations that run locally (duckdb/filesystem/postgres)
**Implementation**
Please do 3 PRs for 3 tasks
destination info contains:
- destination type, name, environment and fingerprint
- destination str() representation (which is display safe)
- local/remote flag
local destination:
for destinations that may run locally (duckdb, postgres, waeviate, quadrant, filesystem etc.) we should start generating fingerprints
- if destination runs locally (always duckdb, `file://` on filesystem, localhost and 127.0.0.1 + ip6 localhost if connection string) a local flag must be set to true in info
- fingerprint must include `.anonymous_id` as used by telemetry and path to file/database when applicable
- `.anonymous_id` should be detached from telemetry code and become independent (maybe part of `paths.py` module)
Contributor guide
Assessment
This issue has not been assessed yet.