Add new auto check that the agent did not modify source tables
- Dominant language
- Python
- Stars
- 123
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
Follow up from https://github.com/dbt-labs/ade-bench/pull/95#issuecomment-3657425597
If an agent solves a task by modifying the source table, it might destroy data along the way. The idea here it to ensure that the sources are untouched after the setup.sh script has run until the start of the testing phase.
In general the workflow will be:
- discover tables in the schema using https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#get_relations_by_pattern-source
- for each of them, fetch their column names using https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#get_filtered_columns_in_relation-source
- get a count of rows and hash of the their values for each table, using something along these lines
```
SELECT
'my_table' AS table_name,
current_timestamp AS snapshot_at,
COUNT(*) AS row_count,
BIT_XOR(HASH(col1, col2, col3 /* etc */))::BIGINT AS table_hash
FROM my_table;
```
(see also audit helper package which has some related art)
- store that into a table, maybe in a separate parquet file or something so the agent doesn't know it exists 🤫
- after the agent has finished its work, re-hash all of the tables defined in the audit table, and compare the results
- fail if any are different
Contributor guide
Assessment
This issue has not been assessed yet.