apache / apache/datafusion-ballista

`ExecutorMetatada` too verbose

Open
#1,851 13 comments 1 reaction 1 assignee Claimed by @sandugood View on GitHub
bug help wanted
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
2d 9m
Merged PRs (30d)
62

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

`ExecutorMetadata` contains `ExecutorSpecification` & `ExecutorOperatingSystemSpecification `

```rust
pub struct ExecutorMetadata {
/// Unique executor identifier.
pub id: String,
/// Hostname or IP address of the executor.
pub host: String,
/// Port number for data transfer.
pub port: u16,
/// Port number for gRPC communication.
pub grpc_port: u16,
/// Resource specification for this executor.
pub specification: ExecutorSpecification,
/// OS and hardware info for this executor
pub os_info: ExecutorOperatingSystemSpecification,
}
```

where `ExecutorOperatingSystemSpecification ` has more data (quite few of them strings

```rust
pub struct ExecutorOperatingSystemSpecification {
/// System name
pub system_name: String,
/// Kernel version
pub kernel_ver: String,
/// OS version
pub os_ver: String,
/// OS version (long)
pub os_ver_long: String,
/// Number of physical cores available on this executor
pub physical_cores: u32,
/// Number of physical disks available on this executor
pub num_disks: u32,
/// Total disk space on this executor, in bytes
pub total_disk_space: u64,
/// Total available disk space on this executor, in bytes
pub total_available_disk_space: u64,
/// Open files limit on this executor
pub open_files_limit: u64,
}
```

problem is that `ExecutorMetadata` is part of `PartitionLocation`

```rust
#[derive(Debug, Clone)]
pub struct PartitionLocation {
/// The source partition ID from the map stage.
pub map_partition_id: usize,
/// The partition identifier.
pub partition_id: PartitionId,
/// Metadata about the executor hosting this partition.
pub executor_meta: ExecutorMetadata,
/// Statistics about the partition data.
pub partition_stats: PartitionStats,
/// shuffle file id
pub file_id: Option,
/// whether this partition uses sort shuffle
pub is_sort_shuffle: bool,
}
```

which represent partition locations and which is going around from/to scheduler and executor, also it might be cached as part of stage informations. which would take a bit of space and might attribute #1836

**Describe the solution you'd like**

Trim down or change information/data structure representing executor metadata,

**Describe alternatives you've considered**

we can either make some fields of executor metadata optional or completely different data structure but we should not share unnecessary data to the executor

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.