tortoise / tortoise/tortoise-orm
Unexpected update query generated when annotate and prefetch_related api used in pipeline
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 516
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 9
Description
Describe the bug
Unexpected update query generated when annotate and prefetch_related api used in pipeline
To Reproduce
async def main():
await Tortoise.init(config=get_database_config())
current_cst_time = pendulum.now('Asia/Shanghai').time()
current = pendulum.now('UTC')
query = PushConfigDB.filter(
status=PushConfigStatus.RUNNING,
dag__target=Target.GROUP,
push_instances__dag_status__not_in=[DAGStatus.INVALID, DAGStatus.FINISHED],
unfinished_count=0
).annotate(
unfinished_count=Count('push_instances__id')
).prefetch_related('dag', 'push_instances')
print(query.sql())
print(query.update(status=PushConfigStatus.FINISHED).sql())
if __name__ == "__main__":
asyncio.run(main())
Expected behavior
The select query is correctted
SELECT `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start`,COUNT(`push_instance`.`id`) `unfinished_count` FROM `push_config` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL) GROUP BY `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start` HAVING COUNT(`push_instance`.`id`)=0
Expected update query, just one implementation
UPDATE `push_config` JOIN (
SELECT `push_config`.`id` FROM `push_config` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL) GROUP BY `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start` HAVING COUNT(`push_instance`.`id`)=0
) AS temp_table ON push_config.id = temp_table.id SET `status`='finished'
Actually behavior
The update query missing the GROUP BY statements
UPDATE `push_config` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` SET `status`=%s WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided async reproduction and comparing the select and update SQL output. Trace the update-query generation for the annotate, aggregation, and prefetch_related combination; it is done when the update preserves the grouped subquery behavior shown in the expected SQL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100