[ECS] [request]: ECS-AGENT: Increase input buffer size in envfile.go
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Customers sometimes need to pass in large environment variables from env files in ECS. ECS-Agent uses bufio to read these values from the files. By default, bufio's maximum buffer is 64kb.
I would like to request to increase this maximum, or make it configurable by the customer.
**Which service(s) is this request for?**
ECS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I have a customer, for example, whose application uses an 80+kb json object for configuration. The pipeline fails because ECS-Agent throws an error for bufio.Scanner() Token too long. The maximum size for a buffer can be set explicitly in the source code. Currently, ECS-Agent uses the default size.
**Are you currently working around this issue?**
Currently, the only workaround is to use a shortened version of the configuration which comes in under the 64kb maximum size. This is not sustainable long-term for my customers.
**Additional context**
Anything else we should know?
**Attachments**
Bufio docs - https://pkg.go.dev/bufio
envfile.go in ecs-agent (see line 560) - https://github.com/aws/amazon-ecs-agent/blob/master/agent/taskresource/envFiles/envfile.go
StackOverflow mentioning how to configure larger buffer - https://stackoverflow.com/questions/21124327/how-to-read-a-text-file-line-by-line-in-go-when-some-lines-are-long-enough-to-ca
Contributor guide
Research direction
Start with ecs-agent's agent/taskresource/envFiles/envfile.go around line 560 and review how bufio.Scanner reads environment-file values. Compare the current behavior with the bufio documentation and determine whether the requested change should use a larger fixed buffer or customer configuration. Done means values larger than 64 KB no longer fail with a Scanner token-too-long error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100