PowerShell / PowerShell/PSResourceGet

`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls?

Open
#1,045 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Area-Find feature_request
Dominant language
C#
Stars
576
Forks
114
Avg merge
1d 2h
Merged PRs (30d)
7

Description

Summary of the new feature / enhancement

Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.

The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):

# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
            (
                $InstalledModules[0 .. 10].ForEach{
                    "Id eq '{0}'" -f $_
                } -join ' or '
            ) + ')&semVerLevel=1.0.0'
)

Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.

Click to view
# String array of modules
$Modules = [string[]](
    'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',')
)


# Microsoft.PowerShell.PSResourceGet
## Get
### Using Cmdlet as is
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique
    )
}
### Using `ForEach-Object -Parallel`
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        $Modules | ForEach-Object -Parallel {
            Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_
        } -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique
    )
}

## Present results
$PSResourceGetResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'Name'
        'Author' = [string] $_.'Author'
        'Version' = [System.Version] $_.'Version'
        'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion'
    }
} | Format-Table -AutoSize



# Bulk PowerShellGallery NuGet API
## Get
Measure-Command -Expression {
    $PageSize = [byte] 30
    $Page = [byte] 1
    $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]](
        $(
            do {
                $FromIndex = [uint16](($Page-1) * $PageSize)
                $ToIndex   = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count')
                Invoke-RestMethod -Method 'Get' -Uri (
                    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
                        (
                            $Modules[$FromIndex .. $ToIndex].ForEach{
                                "Id eq '{0}'" -f $_
                            } -join ' or '
                        ) + ')&semVerLevel=1.0.0'
                )
                $Page++
            }
            until ($ToIndex -ge $InstalledModules.'Count')
        ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique
    )
}

## Present results
$PowerShellGalleryNuGetAPIResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'title'.'#text'
        'Author' = [string] $_.'author'.'name'
        'Version' = [System.Version] $_.'properties'.'Version'
        'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion'
    }
} | Format-Table -AutoSize
Proposed technical implementation details (optional)
  • Use API to call multiple modules at a time?
  • Multithreading? / Jobs?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the Find-PSResource entry point and trace how a string array of module names becomes repository requests. Compare the current per-name behavior with the PowerShell Gallery NuGet example; done means batched lookup returns the same module results with fewer requests and improved timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, powershell
Domain
performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.