Goal: TSN Connect - Analyse Verified Transparency of Data Providers' Data Lineage

Describe the Feature

As per #137, the TSN Stream Standard Spec outlines an approach to enable an oracle mechanism where Data Providers can:

1. Prove requests made to their data sources.
2. Verify this data within the TSN.

The intended outcome is to minimise the need for trust in Data Providers and enhance data quality in the TSN. By guaranteeing the provenance and integrity of data, we can eliminate the need to trust that the Data Provider did not tamper with the data, and verify data quality in the TSN.

Example Implementation

This involves integrating technology within the tsn-data-provider repository that allows Data Providers to generate proofs of authenticity and integrity for their data sourcing.

However, incorporating novel cryptographic primitives into the TSN at this stage may introduce unforeseen risks, such as latency, maintenance concerns, upgradeability, etc. Therefore, we will start with an analysis-first approach.

Initial Phase

Initially, Data Provider technology will be capable of proving data sources. These proofs will be piped to a centralised environment in the Truflation cloud, where they can be verified and analysed. This will provide insights such as:

• The provenance of data.
• The trustworthiness of data sources.
• Differentiation between transparent and opaque data providers.
• The diversity of data sources.

The established analytics data pipeline will serve as a foundation for developing systems focused on:

1. Data provider reputation management.
2. Quality control.
3. Network operator profiling for uptime, health, and optimisation insights.
4. DRM compliance, facilitating actions such as slashing or other legal penalties for data piracy.

Architecture:

Outcome:

This analytical approach will help determine whether we should fully commit to integrating verification directly into TSN Nodes. Additionally, this analysis can be publicly shared to provide the TRUF Nation with visibility into data sources, visualised via a Graph of Entities - including Sources, Data Providers, and TSN Nodes (refer to Grafana Node Graph.

Next Steps

Ryan and Raffael will lead this effort, alongside contributions from Usher Labs. This involves providing the underlying cryptography, technology, and framework for implementation. The dependency repository can be found here: Usher Labs Verity. Although it is private, anyone from TSN can request read access from Ryan, which will be granted.

The development of the analytical pipeline is in progress. Proofs will be piped into Vector, where the Data Provider can collect and manage telemetry internally, as well as syndicate this telemetry to Truflation's own Vector instance. From here, Prometheus and Grafana will be used as the basis for the analytics stack.

@outerlook will support the deployment of this stack once the pipeline is developed.

For now, we can begin by creating the tsn-connect directory within the tsn-data-provider repository to house the Prover instance. The Prover is essentially a local proxy where API requests can be forwarded to automate the processing of cryptographic requirements.

Start by reading issue #137 and the linked TSN Stream Standard Spec, then review the tsn-data-provider repository and its proposed tsn-connect directory. Request access to Usher Labs Verity to understand the Prover dependency. Done means the analysis, integration scope, and requirements for routing proofs into the Vector, Prometheus, and Grafana pipeline are documented.