PowerShell / PowerShell/PSResourceGet
`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls?
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 576
- Forks
- 114
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 7
Description
Summary of the new feature / enhancement
Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.
The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):
# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
(
$InstalledModules[0 .. 10].ForEach{
"Id eq '{0}'" -f $_
} -join ' or '
) + ')&semVerLevel=1.0.0'
)
Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.
Click to view
# String array of modules
$Modules = [string[]](
'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',')
)
# Microsoft.PowerShell.PSResourceGet
## Get
### Using Cmdlet as is
Measure-Command -Expression {
$Script:PSResourceGetResults = [array](
Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique
)
}
### Using `ForEach-Object -Parallel`
Measure-Command -Expression {
$Script:PSResourceGetResults = [array](
$Modules | ForEach-Object -Parallel {
Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_
} -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique
)
}
## Present results
$PSResourceGetResults.ForEach{
[PSCustomObject]@{
'Name' = [string] $_.'Name'
'Author' = [string] $_.'Author'
'Version' = [System.Version] $_.'Version'
'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion'
}
} | Format-Table -AutoSize
# Bulk PowerShellGallery NuGet API
## Get
Measure-Command -Expression {
$PageSize = [byte] 30
$Page = [byte] 1
$Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]](
$(
do {
$FromIndex = [uint16](($Page-1) * $PageSize)
$ToIndex = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count')
Invoke-RestMethod -Method 'Get' -Uri (
'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
(
$Modules[$FromIndex .. $ToIndex].ForEach{
"Id eq '{0}'" -f $_
} -join ' or '
) + ')&semVerLevel=1.0.0'
)
$Page++
}
until ($ToIndex -ge $InstalledModules.'Count')
) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique
)
}
## Present results
$PowerShellGalleryNuGetAPIResults.ForEach{
[PSCustomObject]@{
'Name' = [string] $_.'title'.'#text'
'Author' = [string] $_.'author'.'name'
'Version' = [System.Version] $_.'properties'.'Version'
'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion'
}
} | Format-Table -AutoSize
Proposed technical implementation details (optional)
- Use API to call multiple modules at a time?
- Multithreading? / Jobs?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Find-PSResource entry point and trace how a string array of module names becomes repository requests. Compare the current per-name behavior with the PowerShell Gallery NuGet example; done means batched lookup returns the same module results with fewer requests and improved timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, powershell
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100