TypeInferenceProvider gen_cache() raises `[Errno 7] Argument list too long`
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 229
- PR merge metrics
- No merged PRs in 30d
Description
## Issue
1. Call made to [TypeInferenceProvider.gen_cache()](https://github.com/Instagram/LibCST/blob/c023fa7c4caff3fd2b3946080f9a58b539b10363/libcst/metadata/type_inference_provider.py#L56) with a long list of paths for the `paths` argument.
2. Exception raised during the creation of `subprocess.Popen` [here](https://github.com/Instagram/LibCST/blob/c023fa7c4caff3fd2b3946080f9a58b539b10363/libcst/metadata/type_inference_provider.py#L105), where `command` argument is now a very long string of comma-delimited file paths.
- Exact error message:
```
[Errno 7] Argument list too long: '/bin/sh'
```
- System: Unix
Seems to be related to the system-imposed ARG_MAX limit on POSIX-compatible systems (see [this](https://stackoverflow.com/questions/29801975/why-is-the-subprocess-popen-argument-length-limit-smaller-than-what-the-os-repor) similar issue on SO).
Wondering if it would be reasonable to have `gen_cache` split up the `paths` and call `run_command` in batches so that the `command` string is within the system-imposed ARG_MAX limit? This could also be done from before the call to `gen_cache` of course, but may have value by being integrated into the API?
Caveats
- These limits vary with every system ([chart](https://www.in-ulm.de/~mascheck/various/argmax/#results))
- On Windows, the limitation comes from the shell itself and not the system (see similar SO issue [here](https://stackoverflow.com/questions/2381241/what-is-the-subprocess-popen-max-length-of-the-args-parameter))
Contributor guide
Assessment
This issue has not been assessed yet.