tortoise / tortoise/tortoise-orm

Unexpected update query generated when annotate and prefetch_related api used in pipeline

Open
#1,279 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Describe the bug
Unexpected update query generated when annotate and prefetch_related api used in pipeline

To Reproduce

async def main():
    await Tortoise.init(config=get_database_config())

    current_cst_time = pendulum.now('Asia/Shanghai').time()

    current = pendulum.now('UTC')

    query = PushConfigDB.filter(
        status=PushConfigStatus.RUNNING,
        dag__target=Target.GROUP,
        push_instances__dag_status__not_in=[DAGStatus.INVALID, DAGStatus.FINISHED],
        unfinished_count=0
    ).annotate(
        unfinished_count=Count('push_instances__id')
    ).prefetch_related('dag', 'push_instances')

    print(query.sql())

    print(query.update(status=PushConfigStatus.FINISHED).sql())


if __name__ == "__main__":
    asyncio.run(main())

Expected behavior

The select query is correctted

SELECT `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start`,COUNT(`push_instance`.`id`) `unfinished_count` FROM `push_config` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL) GROUP BY `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start` HAVING COUNT(`push_instance`.`id`)=0

Expected update query, just one implementation

UPDATE `push_config` JOIN (
    SELECT `push_config`.`id` FROM `push_config` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL) GROUP BY `push_config`.`budget`,`push_config`.`id`,`push_config`.`groups`,`push_config`.`effective_duration`,`push_config`.`account_id`,`push_config`.`name`,`push_config`.`receive_user_time_start`,`push_config`.`execution_time_end`,`push_config`.`receive_user_time_end`,`push_config`.`update_time`,`push_config`.`exactly_once_delivery`,`push_config`.`dag_id`,`push_config`.`runtime_params`,`push_config`.`create_time`,`push_config`.`status`,`push_config`.`robots`,`push_config`.`execution_time_start` HAVING COUNT(`push_instance`.`id`)=0
) AS temp_table ON push_config.id = temp_table.id SET `status`='finished'

Actually behavior

The update query missing the GROUP BY statements

UPDATE `push_config` LEFT OUTER JOIN `directed_acyclic_graph` `push_config__dag` ON `push_config__dag`.`id`=`push_config`.`dag_id` LEFT OUTER JOIN `push_instance` ON `push_config`.`id`=`push_instance`.`push_config_id` SET `status`=%s WHERE `push_config`.`status`='running' AND `push_config__dag`.`target`='group' AND (`push_instance`.`dag_status` NOT IN ('invalid','finished') OR `push_instance`.`dag_status` IS NULL)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided async reproduction and comparing the select and update SQL output. Trace the update-query generation for the annotate, aggregation, and prefetch_related combination; it is done when the update preserves the grouped subquery behavior shown in the expected SQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.