performance issues with workflow loop iterations and large amounts of data

SUMMARY

I have a workflow that loops another workflow some number of times (concurrency=2). the child workflow has some fairly linear actions they take.
If i run the child workflow alone, it takes about 90 seconds to complete.
If i run the parent and it runs 2 iterations of the child, each iteration takes about the same.
If it runs with 4 iterations, the first 2 take 90s, the second 2 take 180-205s
5+ iterations: (1-2) -85s, (3-4) -180-205s, (5-8) 200-300s, (9+) -250-350s
on iteration 6, i counted up the actual execution time for all of the actions, each individual action added up had a total execution time of 71s. If i interpret this correctly that means for the 199s this specific iteration ran, only 71s was spent on actual actions.

The observation i have is the more iterations, even with concurrency=2, the longer each subsequent iteration takes.
I can also confirm that there are no other executions running at this time
I have 10 action runners active, on a 4 core VM, and the runners appear pretty well spread out across at least 3 cores at any given time.
CPU utilization is pretty spiky for 1 core at a time, never multiple.

Execution of parent workflow, and execution of one child iteration.
https://pastebin.com/0ksPRYNv

Re-run of the same child shortly after as a single run.
https://pastebin.com/GVjmRxNG

Sanitized child workflow
https://pastebin.com/1GbTyjVN

This is usually ran against a cisco router/switch. and each device can generate between 100~500 'rows' of client data. each 'row' consists of 7 simple (and short) k:v pairs.

STACKSTORM VERSION

st2 --version
st2 3.1.0, on Python 2.7.12

OS, environment, install method

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"

curl -sSL -O https://stackstorm.com/packages/install.sh && chmod +x install.sh
./install.sh --user="${AUTOMATION_USER}" --password="${AUTOMATION_PWD}" --version="${ST2_VERSION}"

Expected Results

For the total run time to be relatively consistent based on the number of loop iterations.

Actual Results

total time for 16 iterations of a 1 minute workflow : 2479s (41 minutes)

i reran one of the iterations as a single, and it took 412s Vs 89s
Same target host, same amount of data.

Troubleshooting

With @m4dcoder 's help, I realized that the over the course of the workflow, I am working with large amounts of data. Furthermore, because of how that large amount of data was being output by actions 3 different times, it appears that there was a compounding effect in the ability to just process what was coming in and out of actions from a raw text standpoint.

By refactoring the WF a tad to remove 1 of these 3, I saw a HUGE performance increase. (2500s -> 290s).

Start by comparing the parent workflow's loop iterations with the same child workflow run alone, using the linked execution data and sanitized workflow. Trace the action-runner and workflow execution timing for repeated large outputs, then verify that runtime no longer compounds across iterations and matches the expected workload.