aws / aws/jsii-rosetta

Inefficient loading of fixtures

Open
#1,986 0 comments 0 reactions 0 assignees View on GitHub
p2
Dominant language
TypeScript
Stars
30
Forks
22
Avg merge
1h 28m
Merged PRs (30d)
44

Description

The change in https://github.com/aws/jsii-rosetta/pull/1811 replaced parallel fixturization (= infusing a snippet into a fixture to convert it from human readable but non-compiling code to less human friendly but compiling code) with a serial approach to limit file system spikes.

As it turns out the loading of fixtures is highly inefficient and involves multiple FS calls per snippet. However there is a 1:N relationship between fixtures and snippets, i.e. a single fixture can be used for multiple snippets. That means the number of FS calls to fixturize snippets can be limited to the number of fixtures.

While in the worst case this can still be equal to the number of snippets, in practice it is unlikely.

We can address this inefficiency by loading all fixtures in serial (which will have FS calls), and then execute fixturization of the snippets in parallel (now without FS calls).

Additionally we can look into limiting the parallelism to a constant load.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the fixture-loading and snippet-fixturization flow described in the issue, and measure its current filesystem calls and parallel work. Done means fixtures are loaded serially once, snippets are fixturized in parallel without repeated filesystem calls, and parallelism is bounded if that investigation supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.