Skip to content

Concurrent installation#1950

Open
alerickson wants to merge 83 commits intomasterfrom
concurrentInstall
Open

Concurrent installation#1950
alerickson wants to merge 83 commits intomasterfrom
concurrentInstall

Conversation

@alerickson
Copy link
Copy Markdown
Member

PR Summary

This PR introduces concurrent (parallel) execution for package find/install workflows in PSResourceGet, shifting portions of the implementation to async + thread-safe data structures so multiple package operations can run at the same time.

High level, it’s aimed at improving performance and throughput when users install or query multiple resources in a single invocation by allowing work to be performed concurrently rather than strictly synchronously.

PR Context

Installing (or resolving) many packages sequentially can be slow, especially when many packages are requested, multiple repositories are queried, or network-bound operations dominate runtime.
To support concurrency safely, the PR also updates internal state handling (caches / hashes) to be thread-safe, preventing race conditions and corruption when multiple tasks update shared structures during parallel execution.

PR Checklist

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

alerickson and others added 2 commits April 27, 2026 19:18
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
@alerickson
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Comment thread src/code/FindHelper.cs
Comment thread src/code/FindHelper.cs
Comment on lines +1117 to +1124
_packagesFound.TryUpdate(foundPkgName, newPkgVersions, pkgVersions);

addedToHash = true;
}
}
else
{
_packagesFound.Add(foundPkg.Name, new List<string> { foundPkgVersion });
_packagesFound.TryAdd(foundPkg.Name, new List<string> { foundPkgVersion });
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can these to operations of TryAdd and TryUpdate be replaced by AddOrUpdate?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TryAdd() and TryUpdate() are ConcurrentDictonary methods. I could write an AddOrUpdate() method if that makes things more clear or simple

Comment thread src/code/FindHelper.cs
Comment on lines +1138 to +1147
if (_knownLatestPkgVersion.ContainsKey(foundPkgName))
{
_knownLatestPkgVersion.TryGetValue(foundPkgName, out PSResourceInfo oldPkgVersion);
_knownLatestPkgVersion.TryUpdate(foundPkgName, foundPkg, oldPkgVersion);
addedToHash = true;
}
else
{
_knownLatestPkgVersion.TryAdd(foundPkg.Name, foundPkg);
addedToHash = true;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider AddOrUpdate

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above

Comment thread src/code/FindHelper.cs
Comment thread src/code/FindHelper.cs
// depPkgsFound creates a new instance of depPkgsFound each time FindDependencyPackages() is called.
// This will eventually return the PSResourceInfo object to the main cmdlet class.
debugMsgs.Enqueue($"Adding'{key}' to list of dependency packages found");
depPkgsFound.TryAdd(key, depPkg);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TryAdd adds if not exist. Maybe dont need to check if it contains it before a TryAdd

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition is there primarily to know whether to call FindDependencyPackagesHelper(), I could move the TryAdd outside of the condition, but I wouldn't be able to get rid of the condition. I figure if the condition is already being checked, it saves some calls to TryAdd if we already know the depPkg is in depPkgsFound

Comment thread src/code/V3ServerAPICalls.cs
Comment thread test/FindPSResourceTests/FindPSResourceV2Server.Tests.ps1
Comment thread test/FindPSResourceTests/FindPSResourceV2Server.Tests.ps1
Comment thread src/code/V2ServerAPICalls.cs
Comment thread src/code/InstallHelper.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants