aws / aws/aws-cli

Compile option to build a single-file executable

Open
#9,149 1 comment 7 reactions 0 assignees View on GitHub
feature-request p3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the feature

When running the AWS CLI on a slow NFS network, any simple command takes a long time, depending on the NFS load.
A strace execution shows about 4000 files are read, mostly from botocore Python library.
The same AWS CLI installed on a local disk can show same execution in 0.5s, which can be 10x or 100x faster than the execution on a slow NFS disk.
There's already the option `--with-install-type=portable-exe` but this still stores the Python libraries in many small files.
The previous feature request #6388 was approximately similar but was solved differently with a containerization.
What I'm looking for is a new compilation option that would trigger the `--onefile` option of the _pyinstaller_ so that the final "aws" executable would require less NFS accesses.

### Use Case

Many small aws commands launched in a shell script makes the overall script slow if each command takes 5s-10s instead of 0.5s with a local disk installation. On the other hand, it's not practical to run local installations on all the hosts of the cluster.

### Proposed Solution

The AWS CLI container is a workaround, but still there's a small overhead to always run a container. When I launch the same command on a slow NFS disk is takes between 5s and 10s, vs. 1.5s when executed from a container, and 0.5s when executed from a local disk installation.

### Other Information

Peformance measurements:
- on a slow NFS installation:
```
/usr/bin/time -p /aws eks get-token --cluster-name mycluster | grep real
real 6.63
real 7.16
real 7.45
```
- on a local disk, same host:
```
/usr/bin/time -p /tmp/aws-cli/v2/2.22.18/bin/aws eks get-token --cluster-name mycluster | grep real
real 0.55
real 0.54
real 0.54
```
- with the aws cli container:
```
/usr/bin/time -p apptainer exec aws.sif aws eks get-token --cluster-name mycluster | grep real
real 1.56
real 1.46
real 1.48
```

Filesystem accesses:
```
% strace -s 200 -f -t /aws eks get-token --cluster-name mycluster | grep real > & strace.log
% grep "open " strace.log | wc -l
3907
```

### Acknowledgements

- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change

### CLI version used

2.22.18

### Environment details (OS name and version, etc.)

RHEL8

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.