Azure / Azure/azure-powershell

Issue with Get-AzDataLakeGen2ChildItem throwing System.OutOfMemoryException

Open
#15,875 3 comments 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad customer-reported Data Lake Storage Gen2 question Service Attention
Dominant language
C#
Stars
4.8k
Forks
4.3k
Avg merge
2d 17h
Merged PRs (30d)
51

Description

Hello,

I'm using `Get-AzDataLakeGen2ChildItem` as recommended in [documentation](https://docs.microsoft.com/en-us/powershell/module/az.storage/get-azdatalakegen2childitem?view=azps-6.4.0#example-3--list-items-recursively-from-a-filesystem-in-multiple-batches) to extract all our paths and ACLs in ADLS. This is the code exactly I'm using:
```
do {
$partItems = Get-AzDataLakeGen2ChildItem -Context $ctx -FileSystem $contName -Recurse -MaxCount $MaxReturn -ContinuationToken $Token
$Total += $partItems.Count
$items += $partItems
if($partItems.Length -le 0) { Break;}
$Token = $partItems[$partItems.Count -1].ContinuationToken;
}
While ($Token -ne $Null)
```

The issue is that without ContinuationToken it returns only small part, but when using ContinuationToken it always fails with System.OutOfMemoryException on the line with Get-AzDataLakeGen2ChildItem, even when I excluded directories, tried with filtering of results etc.

Is there any way to improve it and be able to extract/process all items in the container for ADLS? Or is it Powershell limitation?

Contributor guide

Open the contributing guide

Research direction

Start with the documented Get-AzDataLakeGen2ChildItem recursive batching example and reproduce the supplied loop against ADLS, comparing calls with and without ContinuationToken. Trace where the cmdlet handles -MaxCount, continuation tokens, directories, and result filtering. Done means all filesystem paths and ACLs can be processed without System.OutOfMemoryException.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.