Goal: Facilitate Pipeline Creation

There are some repetitive parts of a general pipeline which could be further improved.

Right now, we provide building blocks for different ingestion structures. But we could further improve it with the correct abstractions on the pipeline execution. Examples:

• Data reconciliation strategies
  • By last inserted (example)
  • Full compare
  • Gap filling
  • By key
  • ...
• Data Source abstract class
  • list_all
  • get_all
  • get_by_date
  • get_by_key (example: https://github.com/trufnetwork/adapters/blob/main/src/tsn_adapters/tasks/argentina/provider.py#L23 but a lot to improve)
• Transformer abstract class to prepare for TN (example)

and it could be well structured to require only the correct methods depending on the strategies. The intention is to create a well-tested generic structure that makes it easier to include new adapters

Start by reading the linked reconciliation.py, provider.py, and transformers.py examples to understand the current adapter patterns. Then define the generic pipeline, data-source, reconciliation, and transformer abstractions and determine the strategy-specific methods they require. Done would require an agreed design, implementation, and comprehensive tests for the supported ingestion strategies.