dbt-labs / dbt-labs/ade-bench

Add new auto check that the agent did not modify source tables

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
123
Forks
25
PR merge metrics
No merged PRs in 30d

Description

Follow up from https://github.com/dbt-labs/ade-bench/pull/95#issuecomment-3657425597

If an agent solves a task by modifying the source table, it might destroy data along the way. The idea here it to ensure that the sources are untouched after the setup.sh script has run until the start of the testing phase.

In general the workflow will be:
- discover tables in the schema using https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#get_relations_by_pattern-source
- for each of them, fetch their column names using https://github.com/dbt-labs/dbt-utils?tab=readme-ov-file#get_filtered_columns_in_relation-source
- get a count of rows and hash of the their values for each table, using something along these lines
```
SELECT
'my_table' AS table_name,
current_timestamp AS snapshot_at,
COUNT(*) AS row_count,
BIT_XOR(HASH(col1, col2, col3 /* etc */))::BIGINT AS table_hash
FROM my_table;
```

(see also audit helper package which has some related art)
- store that into a table, maybe in a separate parquet file or something so the agent doesn't know it exists 🤫
- after the agent has finished its work, re-hash all of the tables defined in the audit table, and compare the results
- fail if any are different

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.