Source Mapper Improvements: Better Performance and Its Consequences
- 主要言語
- Python
- スター
- 288
- フォーク
- 138
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
## Problem
The source mapper is very slow and a memory hog. As a result, two non-ideal artifacts have been added:
1. `pyteal.ini` which defaults to `pyteal-source-mapper.enabled = False`
2. Nightly tests in C.I. which carve out the slowest portion of the tests and complicate local testing
## Solution
@jasonpaulos has an [experimental approach](https://github.com/jasonpaulos/pyteal/tree/source-map-performance) which promises to speed things up to the point that these artifacts can be eliminated, or mitigated.
I propose:
* nightly tests be eliminated once it is established that all tests can be run in a reasonable amount of time (however #630 should be taken into account before eliminating)
* Default the sourcemapper to `enabled = True` in the `pyteal.ini` but don't get rid of it
### Why keep the `pyteal.ini` ?
* It may still be useful to allow dev's to turn the feature off completely.
* We may desire to feature gate (cf. #601 ) via `pyteal.ini`. However, a more modern approach uses `pyproject.toml` instead.
## Tasks
- [ ] Improvements to `NatalStackFrame` and `StackFrame`
- [ ] only keep the "best" frame, not a list of best frames (which in actuality always has length 1)
- [ ] incorporate ideas from @jasonpaulos [branch](https://github.com/jasonpaulos/pyteal/tree/source-map-performance) and this [S.O. post](https://stackoverflow.com/questions/17407119/python-inspect-stack-is-slow)
- [ ] investigate saving memory by using __slots__ (e.g.: [nice example](https://khuyentran1401.github.io/Efficient_Python_tricks_and_tools_for_data_scientists/Chapter1/class.html#optimizing-memory-usage-in-python-with-slots) )
- [ ] investigate replacing the debug-only member `StackFrame.full_stack` by `StackFrame._debug_frame_origin` and possibly add a method `StackFrame._debug_full_stack()` which can recreate this information
- [ ] modify the unit tests that employ `NatalStackFrame._keep_all_debugging` so that they can work on the new lighter frame without keeping the entire stack. These are the following:
- [ ] `tests/unit/sourcemap_test.py::test_frames`
- [ ] `tests/unit/sourcemap_monkey_unit_test.py::test_frame_info_is_right_before_core_last_drop_idx`
- [ ] Nightly tests
- [ ] bring back python 3.10 source map tests into the "On Commit" workflow
- [ ] either remove the nightly workflow, or repurpose it
- [x] `pyteal.ini` (addressed in #687 and #688)
- [x] decide whether to keep it and/or use `pyproject.toml` instead (DECISION: no file-based configs)
- [x] if keeping, default source mapping to `enabled`
- [x] or maybe support both: Adopt [pytest's approach](https://github.com/pytest-dev/pytest/blob/857e34ef8555c48cb5c44f143a0d6692efb6c60f/src/_pytest/config/findpaths.py#L66) for dealing with both `*.ini` and `pyproject.toml` files. In particular, in the case that `pyteal.ini` isn't found, look also for `pyproject.toml`, and then for the section `[tool.pyteal.ini]` Such a section should look like:
```toml
[tool.pyteal.ini]
source-mapper-enabled = true
source-mapper-debug = false
```
## Dependencies
None
## Urgency
Medium - defaulting to `pyteal-source-mapper.enabled = True` will make source mapping a lot more accessible, and getting rid of nightly tests will reduce developer friction.
コントリビューションガイド
評価
この issue はまだ評価されていません。