Azure / Azure/azure-powershell
Issue with Get-AzDataLakeGen2ChildItem throwing System.OutOfMemoryException
- Dominant language
- C#
- Stars
- 4.8k
- Forks
- 4.3k
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 51
Description
Hello,
I'm using `Get-AzDataLakeGen2ChildItem` as recommended in [documentation](https://docs.microsoft.com/en-us/powershell/module/az.storage/get-azdatalakegen2childitem?view=azps-6.4.0#example-3--list-items-recursively-from-a-filesystem-in-multiple-batches) to extract all our paths and ACLs in ADLS. This is the code exactly I'm using:
```
do {
$partItems = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $contName -Recurse -MaxCount $MaxReturn -ContinuationToken $Token
$Total += $partItems.Count
$items += $partItems
if($partItems.Length -le 0) { Break;}
$Token = $partItems[$partItems.Count -1].ContinuationToken;
}
While ($Token -ne $Null)
```
The issue is that without ContinuationToken it returns only small part, but when using ContinuationToken it always fails with System.OutOfMemoryException on the line with Get-AzDataLakeGen2ChildItem, even when I excluded directories, tried with filtering of results etc.
Is there any way to improve it and be able to extract/process all items in the container for ADLS? Or is it Powershell limitation?
Contributor guide
Research direction
Start with the documented Get-AzDataLakeGen2ChildItem recursive batching example and reproduce the supplied loop against ADLS, comparing calls with and without ContinuationToken. Trace where the cmdlet handles -MaxCount, continuation tokens, directories, and result filtering. Done means all filesystem paths and ACLs can be processed without System.OutOfMemoryException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, powershell
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100