ADF bootsrapping pipeline taks long time to compelete

When provisioning new accounts with ADF, we run the bootstrap pipeline. Currently, our pipeline takes around half an hour, and as the number of accounts increases, the running time also increases linearly, therefore, we have the concern that it takes too long and may decrease our producivity. For example, if the number of our accounts goes 10 times more, then it would take around 5 hours per run, which is too long.

Another more important reason is that we also do some customizations in some python code such as the 'generate_param.py' in the shared folder. And testing/debugging such change would be inconvenient for us if the pipeline takes more than half an hour to run. So I wonder whether there is already something that can make the bootstrap pipeline run faster?

A raw idea in my mind is that we could probably keep the status of the adf.yml, and we only provision the changes in this file? Not sure whether you already have something similar, or maybe you have some clients that have a similar issue and already have solutions for them?

Start by tracing the bootstrap pipeline and reading adf.yml alongside the shared generate_param.py customization mentioned in the issue. Measure where runtime grows with account count and check whether the current pipeline already preserves provisioning state or supports incremental changes. Done should be a defined, validated approach that reduces repeat bootstrap time without skipping required account provisioning.

Written by the indexing model from the issue text.