Instagram / Instagram/LibCST

TypeInferenceProvider gen_cache() raises `[Errno 7] Argument list too long`

Open
#332 2 comments 0 reactions 0 assignees View on GitHub
codemod
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

## Issue
1. Call made to [TypeInferenceProvider.gen_cache()](https://github.com/Instagram/LibCST/blob/c023fa7c4caff3fd2b3946080f9a58b539b10363/libcst/metadata/type_inference_provider.py#L56) with a long list of paths for the `paths` argument.
2. Exception raised during the creation of `subprocess.Popen` [here](https://github.com/Instagram/LibCST/blob/c023fa7c4caff3fd2b3946080f9a58b539b10363/libcst/metadata/type_inference_provider.py#L105), where `command` argument is now a very long string of comma-delimited file paths.
- Exact error message:
```
[Errno 7] Argument list too long: '/bin/sh'
```
- System: Unix

Seems to be related to the system-imposed ARG_MAX limit on POSIX-compatible systems (see [this](https://stackoverflow.com/questions/29801975/why-is-the-subprocess-popen-argument-length-limit-smaller-than-what-the-os-repor) similar issue on SO).

Wondering if it would be reasonable to have `gen_cache` split up the `paths` and call `run_command` in batches so that the `command` string is within the system-imposed ARG_MAX limit? This could also be done from before the call to `gen_cache` of course, but may have value by being integrated into the API?

Caveats
- These limits vary with every system ([chart](https://www.in-ulm.de/~mascheck/various/argmax/#results))
- On Windows, the limitation comes from the shell itself and not the system (see similar SO issue [here](https://stackoverflow.com/questions/2381241/what-is-the-subprocess-popen-max-length-of-the-args-parameter))

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.