target-datalake-anywhere

Inspired by tap-spreadsheets-anywhere... but focused on the target side and some common "data lake" target requirements:

Generic File Target Capabilities

1. Serialize data in one of many common interop formats, especially: csv, parquet, jsonl, and jsonl.gz
2. Store data locally or leverage one of the mainstream blob storage providers: AWS S3, Azure ADLS, or Google Storage
3. Support some kind of 'merge upsert' capability, if possible, or else communicate to the user that writes are in "append-mode" and won't override prior versions of records.

Advanced "Data Lake" Capabilities

And as this target matures, presumably it could add some extensions to handle common challenges for targets of the 'data lake' type:

1. If locking is needed to protect data integrity, provide some mechanism to lock datasets while they are being written to.
2. When locking is not possible, provide atomic 'swap' capability between old and new versions of datasets.
   • Possible implementation:
     1. First write out all new dataset files without modifying the prior versions.
     2. Then as a single ACID action or "transaction", upload a manifest file which then points consumers to the updated datasets.
     3. Optionally purge older stale copied of datasets - but lazily after some time has passed and we are sure no consumers are still reading from them.
3. Especially when writing to file formats which are not inherently self describing (such as CSV), have an option to leave behind some metadata for the consuming process to leverage - such as a JSON schema describing the properties and schema of each "table".

Implementation

There are many ways to approach this, but a couple patterns seem worth suggesting here:

1. As a base interop layer, smart_open or PyFilesystem could provide generic write capabilities to n number of clouds and data formats.
2. DuckDB is pip installable and could be leveraged for advanced merge upserts. Alternatively, a library such as pandas could provide similar generic 'merge upsert' capabilities, with the caveat that all data must fit into memory on the runner.

The issue names no repository files, tests, or entry points. Start by examining the repository structure and existing target implementations, then narrow the broad storage, format, merge, locking, and metadata requirements to a defined first milestone with corresponding tests and acceptance criteria.