[BUG] file.append can be extremely slow with just tens of lines
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
The below simple setup has 50% chance to take 10+ seconds when applying several times. The time increases exponentially with the number of lines in the file.
Setup
# test.sls
/tmp/test.txt:
file.append:
- source: salt://test.txt
# test.txt
{%- for ip in range(25) -%}
a
{% endfor -%}
{{ 2 | random_hash }}
Steps to Reproduce the behavior
for i in {1..10}; do salt-call --log-level=debug state.apply test; done
Expected behavior
The command should be finished within tens of ms.
Versions Report
v2019.2.3. Saw similar on v2018.3.2 as well before upgrading to v2019.2.3.
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 2019.2.3
Dependency Versions:
cffi: 1.14.0
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.6.2
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Oct 8 2019, 14:14:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.15.0-96-generic
system: Linux
version: Ubuntu 16.04 xenial
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the test.sls and test.txt reproducer, then run the provided salt-call loop with debug logging on Salt 2019.2.3 or a comparable environment. Trace the file.append execution and templated source handling to identify the scaling delay; done means repeated applies complete within tens of milliseconds for the sample input without changing its result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100