matrixorigin / matrixorigin/matrixone

[Tech Request]: Intelligent Query Execution Scheduling

Open
#25,451 1 comment 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
kind/tech-request
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same tech request?

- [x] I have checked the existing issues.

### Does this tech request not affect user experience?

- [x] This tech request doesn't affect user experience.

### What would you like to be added ?

```Markdown
At this moment, MO will parse a query and based on some logic to determine if the query is a AP query or TP query. TP query is executed on the current CN (handling connection), and AP will be executed on ALL available CN (and with all threads on each CN).

This is unreasonably restrictive. In theory, a query can be executed on ANY or ANY subset of CNs, and especially, the subset of CNs does not need to include current CN (client is connected)
```

### Why is this needed ?

To utilize hardware info much more efficiently.

### Additional information

There is a routing decision that should be made after query parsing. It could be automatic, intelligent, for example, look at ALL CNs, pick up CNs with load less than 50%. Use this subset of CN.

Or a more manual style. For example, we can give each CN a tag, so a cluster may have say, a few TP CNs, a few ETL CNs, and some AP CNs. Let's assume clients are all connected to TP CNs. After parsing, we can execute all TP queries in current CN. For LOAD, etc, run the work on the ETL CNs, and for large AP queries, send to those AP CNs. These different CNs (TP, LOAD, AP) does not need to have same hardware resource.
We can optimize allocation, for example, more memory for TP CNs and less memory, but more disk bandwidth for ETL CNs, etc.

One interesting case, is that we can have many customers, each have its own TP CN, let's assume each customer has just one TP CN. A shared pool of AP CNs for all customers to run AP queries. Note that this case the AP query is executed on a subset of CNs that does not include current CN.

Also on TPC-C like workload, we can have a bunch of CNs, based on the warehouse id after query parsing, we either execute the query on current CN, or, route the the CN that "owns" the partition. If we do this, TPC-C does not have any distributed transactions. This is actually quite common because some experienced DBA will shard data exactly to avoid transactions across shards.

Finally, I do not know if our AP queries runs correctly on several CNs if each of them are configured with different number of CPUs. And I don't think we can run a query only use a subset of the CPUs on a CN -- that is, if we can run a pipeline with parallelism 4 even if the machine has 8 cores. But these may not as important if we fully embraces K8S -- we can always partition a bigger machine to a standard "unit" size.
But worth find out where we fail right now and understand the problems at least.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.