argotorg / argotorg/sourcify

Self-contained parquet export

Open
#2,500 0 comments 0 reactions 1 assignee Claimed by @manuelwedler View on GitHub
database verifier alliance
Dominant language
TypeScript
Stars
949
Forks
542
Avg merge
2d 19h
Merged PRs (30d)
18

Description

> Following #1668 I think we can have a separate export where sources are joined before the export and each partition is made of "self-contained" data. E.g. under `/1/0xab/0xab.part1.parquet` you'll find the columns of this contract joined on the `contract_deployments` table. So that someone can fetch a single row of this partition, that is all information about this contract, with a parquet tool, or just the whole partition.

_Originally posted by @kuzdogan in [#2441](https://github.com/argotorg/sourcify/issues/2441#issuecomment-3450021355)_

As the comment above suggests, we should consider an additional export, where each file contains all data related to the included contracts. This should make it possible to get all data for one contract without having to join over multiple parquet files.

Also see https://github.com/verifier-alliance/parquet-export/issues/10

### Specification

- Parquet files should be split using hive-style partitioning like in this example: `chain_id=1/address_prefix=0xab/1_0xab_part0.parquet`.
- The export joins all relevant columns from our main db tables into each file, meaning that one row contains all data for one verified contract.
- For joining on the many-to-many relationships, we need to aggregate data in a special column. This counts for the `sources` and `signatures` tables. Parquet has native support for [nested structured data](https://arrow.apache.org/blog/2022/10/08/arrow-parquet-encoding-part-2/). I'm still figuring out if this or a json column is best for aggregation.
- Incremental exports: we can make also this export append-only and only rewrite the latest file per partition folder. For this, export queries should use `order by verified_contracts.created_at`. Note that a `contract_deployment` might be exported more than one time when match upgrades happen. This is not a problem since the latest version of a contract can be selected by the `created_at` timestamp.
- Our export script can be refactored to make some common functionalities reusable. We can make it support an argument that determines if classic or hive format should be exported. GCP jobs should run separately for the two formats.
- Deployment:
- We should create a separate bucket on GCP for this.
- The export can be made public at `hive-export.sourcify.dev`.
- Enable the S3-compatible API like in the other export.
- I would also deploy this export for VerA. This would actually for the first time enable to publicly query specific contracts from the VerA db. Not as good as a proper API but still an improvement compared to downloading the complete dataset.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.