PowerShell / PowerShell/PSResourceGet

`Find-PSResource` with a string array of module names is slow: Bulk/batch the API calls?

Aperta
#1,045 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Area-Find feature_request
Lingua principale
C#
Stelle
576
Fork
114
Merge medio
1g 2h
PR unite (30g)
7

Descrizione

Summary of the new feature / enhancement

Doing PowerShellGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $InstalledModules is slow, seems like it queries one at a time.

The NuGet API can do multiple, like for instance (i know latest beta went away from using NuGet):

# Faster, multiple at a time
## Two manually
Invoke-RestMethod -Method 'Get' -Uri "https://www.powershellgallery.com/api/v2/Packages?`$filter=IsLatestVersion and IsPrerelease eq false and (Id eq 'Az.Accounts' or Id eq 'Microsoft.Graph.Authentication')&semVerLevel=1.0.0"
## Ten first from a string array of module names
Invoke-RestMethod -Method 'Get' -Uri (
    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
            (
                $InstalledModules[0 .. 10].ForEach{
                    "Id eq '{0}'" -f $_
                } -join ' or '
            ) + ')&semVerLevel=1.0.0'
)

Here are some PowerShell to show how slow it is currently. I copied out a string array of module names.

Click to view
# String array of modules
$Modules = [string[]](
    'AIPService,AWSPowerShell.NetCore,Az,Az.Accounts,Az.ADDomainServices,Az.Advisor,Az.Aks,Az.AlertsManagement,Az.AnalysisServices,Az.ApiManagement,Az.App,Az.AppConfiguration,Az.ApplicationInsights,Az.ApplicationMonitor,Az.Attestation,Az.Automanage,Az.Automation,Az.BareMetal,Az.Batch,Az.Billing,Az.BillingBenefits,Az.Blueprint,Az.BootStrapper,Az.BotService,Az.Cdn,Az.ChangeAnalysis,Az.CloudService,Az.CognitiveServices,Az.Communication,Az.Compute,Az.Compute.ManagedService,Az.ConfidentialLedger,Az.Confluent,Az.ConnectedKubernetes,Az.ConnectedMachine,Az.ConnectedNetwork,Az.ContainerInstance,Az.ContainerRegistry,Az.CosmosDB,Az.CostManagement,Az.CustomLocation,Az.CustomProviders,Az.Dashboard,Az.DataBox,Az.DataBoxEdge,Az.Databricks,Az.Datadog,Az.DataFactory,Az.DataLakeAnalytics,Az.DataLakeStore,Az.DataMigration,Az.DataProtection,Az.DataShare,Az.DedicatedHsm,Az.DeploymentManager,Az.DesktopVirtualization,Az.DeviceProvisioningServices,Az.DeviceUpdate,Az.DevSpaces,Az.DevTestLabs,Az.DigitalTwins,Az.DiskPool,Az.Dns,Az.DnsResolver,Az.DynatraceObservability,Az.EdgeOrder,Az.Elastic,Az.ElasticSan,Az.EventGrid,Az.EventHub,Az.FluidRelay,Az.FrontDoor,Az.Functions,Az.GuestConfiguration,Az.HanaOnAzure,Az.HDInsight,Az.HealthBot,Az.HealthcareApis,Az.HPCCache,Az.ImageBuilder,Az.ImportExport,Az.Insights,Az.IotCentral,Az.IotHub,Az.KeyVault,Az.KubernetesConfiguration,Az.Kusto,Az.LabServices,Az.LoadTesting,Az.LogicApp,Az.Logz,Az.MachineLearning,Az.MachineLearningCompute,Az.MachineLearningServices,Az.Maintenance,Az.ManagedServiceIdentity,Az.ManagedServices,Az.ManagementPartner,Az.Maps,Az.MariaDb,Az.Marketplace,Az.MarketplaceOrdering,Az.Media,Az.Migrate,Az.MixedReality,Az.MobileNetwork,Az.Monitor,Az.MonitoringSolutions,Az.MySql,Az.NetAppFiles,Az.Network,Az.NetworkFunction,Az.Nginx,Az.NotificationHubs,Az.OperationalInsights,Az.Orbital,Az.Peering,Az.PolicyInsights,Az.Portal,Az.PostgreSql,Az.PowerBIEmbedded,Az.PrivateDns,Az.Profile,Az.ProviderHub,Az.Purview,Az.Quota,Az.RecoveryServices,Az.RedisCache,Az.RedisEnterpriseCache,Az.Relay,Az.Reservations,Az.ResourceGraph,Az.ResourceMover,Az.Resources,Az.Search,Az.Security,Az.SecurityInsights,Az.ServiceBus,Az.ServiceFabric,Az.ServiceLinker,Az.SignalR,Az.SpringCloud,Az.Sql,Az.SqlVirtualMachine,Az.Ssh,Az.StackEdge,Az.StackHCI,Az.StackHCI.NetworkHUD,Az.Storage,Az.StorageMover,Az.StorageSync,Az.StreamAnalytics,Az.Subscription,Az.Support,Az.Synapse,Az.Tags,Az.TimeSeriesInsights,Az.Tools.Installer,Az.Tools.Migration,Az.Tools.Predictor,Az.TrafficManager,Az.VMware,Az.VoiceServices,Az.Websites,Az.WindowsIotServices,AzSK,AzSK.AAD,AzSK.ADO,AzSK.AzureDevOps,Azure,Azure.AnalysisServices,Azure.Storage,AzureAD,AzureADPreview,AzureRM.profile,AzViz,ConfluencePS,DefenderMAPS,Evergreen,ExchangeOnlineManagement,GetBIOS,ImportExcel,Intune.USB.Creator,IntuneBackupAndRestore,Invokeall,JWTDetails,Mailozaurr,Microsoft.Graph,Microsoft.Graph.Applications,Microsoft.Graph.Authentication,Microsoft.Graph.Bookings,Microsoft.Graph.Calendar,Microsoft.Graph.ChangeNotifications,Microsoft.Graph.CloudCommunications,Microsoft.Graph.Compliance,Microsoft.Graph.CrossDeviceExperiences,Microsoft.Graph.DeviceManagement,Microsoft.Graph.DeviceManagement.Actions,Microsoft.Graph.DeviceManagement.Administration,Microsoft.Graph.DeviceManagement.Enrolment,Microsoft.Graph.DeviceManagement.Functions,Microsoft.Graph.Devices.CloudPrint,Microsoft.Graph.Devices.CorporateManagement,Microsoft.Graph.Devices.ServiceAnnouncement,Microsoft.Graph.DirectoryObjects,Microsoft.Graph.Education,Microsoft.Graph.Files,Microsoft.Graph.Financials,Microsoft.Graph.Groups,Microsoft.Graph.Identity.DirectoryManagement,Microsoft.Graph.Identity.Governance,Microsoft.Graph.Identity.SignIns,Microsoft.Graph.Intune,Microsoft.Graph.Mail,Microsoft.Graph.ManagedTenants,Microsoft.Graph.Notes,Microsoft.Graph.People,Microsoft.Graph.PersonalContacts,Microsoft.Graph.Planner,Microsoft.Graph.Reports,Microsoft.Graph.SchemaExtensions,Microsoft.Graph.Search,Microsoft.Graph.Security,Microsoft.Graph.Sites,Microsoft.Graph.Teams,Microsoft.Graph.Users,Microsoft.Graph.Users.Actions,Microsoft.Graph.Users.Functions,Microsoft.Graph.WindowsUpdates,Microsoft.Online.SharePoint.PowerShell,Microsoft.PowerShell.ConsoleGuiTools,Microsoft.PowerShell.SecretManagement,Microsoft.PowerShell.SecretStore,Microsoft.RDInfra.RDPowershell,Microsoft.RdInfra.RDPowershell.Migration,MicrosoftGraphSecurity,MicrosoftPowerBIMgmt,MicrosoftPowerBIMgmt.Admin,MicrosoftPowerBIMgmt.Capacities,MicrosoftPowerBIMgmt.Data,MicrosoftPowerBIMgmt.Profile,MicrosoftPowerBIMgmt.Reports,MicrosoftPowerBIMgmt.Workspaces,MicrosoftTeams,MSAL.PS,MSGraphFunctions,MSOnline,Nevergreen,newtonsoft.json,Office365DnsChecker,Optimized.Mga,Optimized.Mga.AzureAD,Optimized.Mga.Mail,Optimized.Mga.Report,Optimized.Mga.SharePoint,PackageManagement,PartnerCenter,PartnerCenter.NetCore,Pester,platyPS,PnP.PowerShell,PolicyFileEditor,PoshRSJob,PowerShellGet,PSGraph,PSIntuneAuth,PSPackageProject,PSPKI,PSReadLine,PSScriptAnalyzer,PSWindowsUpdate,RunAsUser,SetBIOS,SharePointPnPPowerShellOnline,SHiPS,SpeculationControl,Trackyon.Utils,VSTeam,WindowsAutoPilotIntune'.Split(',')
)


# Microsoft.PowerShell.PSResourceGet
## Get
### Using Cmdlet as is
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $Modules | Sort-Object -Property 'Name' -Unique
    )
}
### Using `ForEach-Object -Parallel`
Measure-Command -Expression {
    $Script:PSResourceGetResults = [array](
        $Modules | ForEach-Object -Parallel {
            Microsoft.PowerShell.PSResourceGet\Find-PSResource -Type 'Module' -Repository 'PSGallery' -Name $_
        } -ThrottleLimit 50 | Sort-Object -Property 'Name' -Unique
    )
}

## Present results
$PSResourceGetResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'Name'
        'Author' = [string] $_.'Author'
        'Version' = [System.Version] $_.'Version'
        'NormalizedVersion' = [System.Version] $_.'AdditionalMetadata'.'NormalizedVersion'
    }
} | Format-Table -AutoSize



# Bulk PowerShellGallery NuGet API
## Get
Measure-Command -Expression {
    $PageSize = [byte] 30
    $Page = [byte] 1
    $Script:PowerShellGalleryNuGetAPIResults = [PSCustomObject[]](
        $(
            do {
                $FromIndex = [uint16](($Page-1) * $PageSize)
                $ToIndex   = [uint16]($Page * $PageSize -lt $Modules.'Count' ? $Page * $PageSize : $Modules.'Count')
                Invoke-RestMethod -Method 'Get' -Uri (
                    'https://www.powershellgallery.com/api/v2/Packages?$filter=IsLatestVersion and IsPrerelease eq false and (' +
                        (
                            $Modules[$FromIndex .. $ToIndex].ForEach{
                                "Id eq '{0}'" -f $_
                            } -join ' or '
                        ) + ')&semVerLevel=1.0.0'
                )
                $Page++
            }
            until ($ToIndex -ge $InstalledModules.'Count')
        ) | Sort-Object -Property @{'Expression'={$_.'title'.'#text'}} -Unique
    )
}

## Present results
$PowerShellGalleryNuGetAPIResults.ForEach{
    [PSCustomObject]@{
        'Name' = [string] $_.'title'.'#text'
        'Author' = [string] $_.'author'.'name'
        'Version' = [System.Version] $_.'properties'.'Version'
        'NormalizedVersion' = [System.Version] $_.'properties'.'NormalizedVersion'
    }
} | Format-Table -AutoSize
Proposed technical implementation details (optional)
  • Use API to call multiple modules at a time?
  • Multithreading? / Jobs?

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Parti dal punto di ingresso Find-PSResource e segui come un array di stringhe di nomi di moduli diventa richieste al repository. Confronta il comportamento attuale per nome con l’esempio NuGet di PowerShell Gallery; il lavoro è completato quando la ricerca in batch restituisce gli stessi risultati dei moduli con meno richieste e tempi migliorati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
csharp, powershell
Ambito
performance, tooling
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
30/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.