apache / apache/datafusion-ballista
`ExecutorMetatada` too verbose
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 2d 9m
- Merged PRs (30d)
- 62
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
`ExecutorMetadata` contains `ExecutorSpecification` & `ExecutorOperatingSystemSpecification `
```rust
pub struct ExecutorMetadata {
/// Unique executor identifier.
pub id: String,
/// Hostname or IP address of the executor.
pub host: String,
/// Port number for data transfer.
pub port: u16,
/// Port number for gRPC communication.
pub grpc_port: u16,
/// Resource specification for this executor.
pub specification: ExecutorSpecification,
/// OS and hardware info for this executor
pub os_info: ExecutorOperatingSystemSpecification,
}
```
where `ExecutorOperatingSystemSpecification ` has more data (quite few of them strings
```rust
pub struct ExecutorOperatingSystemSpecification {
/// System name
pub system_name: String,
/// Kernel version
pub kernel_ver: String,
/// OS version
pub os_ver: String,
/// OS version (long)
pub os_ver_long: String,
/// Number of physical cores available on this executor
pub physical_cores: u32,
/// Number of physical disks available on this executor
pub num_disks: u32,
/// Total disk space on this executor, in bytes
pub total_disk_space: u64,
/// Total available disk space on this executor, in bytes
pub total_available_disk_space: u64,
/// Open files limit on this executor
pub open_files_limit: u64,
}
```
problem is that `ExecutorMetadata` is part of `PartitionLocation`
```rust
#[derive(Debug, Clone)]
pub struct PartitionLocation {
/// The source partition ID from the map stage.
pub map_partition_id: usize,
/// The partition identifier.
pub partition_id: PartitionId,
/// Metadata about the executor hosting this partition.
pub executor_meta: ExecutorMetadata,
/// Statistics about the partition data.
pub partition_stats: PartitionStats,
/// shuffle file id
pub file_id: Option,
/// whether this partition uses sort shuffle
pub is_sort_shuffle: bool,
}
```
which represent partition locations and which is going around from/to scheduler and executor, also it might be cached as part of stage informations. which would take a bit of space and might attribute #1836
**Describe the solution you'd like**
Trim down or change information/data structure representing executor metadata,
**Describe alternatives you've considered**
we can either make some fields of executor metadata optional or completely different data structure but we should not share unnecessary data to the executor
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.