apache / apache/datafusion

[DISCUSS] Distinguish between "performance" and "efficiency" cores

Open
#16,490 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Modern Macs / Apple Silicon (and Intel 12th gen processors) make a distinction between Performance (`P`) and Efficiency (`E`) cores.

https://developer.apple.com/news/?id=vk3m204o

> Recent Apple Silicon like A13 Bionic has both high-performance cores (P cores) and high-efficiency cores (E cores). These different core types allow you to deliver apps that have both great performance and great battery life. To take full advantage of their performance and efficiency, you can provide the operating system (OS) with information about how to execute your app in the most optimal way. From there, the OS uses semantic information to make better scheduling and performance control decisions.

Intel 12th gen chips have this feature too, for example:
https://www.intel.com/content/www/us/en/gaming/resources/how-hybrid-design-works.html

> Intel® Core™ desktop processors integrate two types of cores into a single die: powerful Performance-cores (P-cores) and flexible Efficient-cores (E-cores). Both types of core have a different role.

DataFusion currently evenly partitions work between cores, and does not distinguish or take advantage of the different kinds of cores. This could lead to less than optimal throughput as the (slower) efficiency cores would be given as much work as the faster power cores, leading to the power cores being idle sometimes

@pepijnve may be seen this manifesting as inconsistent behavior / profiling on https://github.com/apache/datafusion/pull/16398

### Describe the solution you'd like

Ensure DataFusion runs on architectures with this hybrid `P`/`E` design efficiently.

I don't know what this means exactly

### Describe alternatives you've considered

Note sure. Maybe we can follow the ideas from Mac:
- https://developer.apple.com/news/?id=vk3m204o

We may also have to have support in tokio 🤔

### Additional context

@pepijnve brought this up on a PR where we are seeing performance variability
- https://github.com/apache/datafusion/pull/16398#issuecomment-2988045575

> 🤔 now that I write '10-core MacBook', I'm wondering if the 10 core part is where my variability is coming from. That's 6 performance and 4 efficiency cores. Ideally DataFusion keeps the CPU bound work on the perf cores, and uses the efficiency ones for IO. I had been wondering about that and NUMA effects already. A topic for a different thread though.

Contributor guide

Open the contributing guide

Research direction

Start by reading the discussion in pull request 16398 and its linked profiling context, then trace how DataFusion partitions work across cores. The issue does not name source files or tests; done would require an agreed approach for hybrid P/E-core scheduling and evidence that CPU-bound and I/O work behave appropriately on those architectures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.