Wrong dependency resolution plus performance optimizations when building (#3757)

Two new parameter in Sort-AppFoldersByDependencies -
onlyTheseAppFoldersPlusDepending and SkipApps. If specified, then only
the appFolders specified in the first array plus the apps depending on
these recursively will be sorted. The remaining folder not included in
the sorted appfolders list will be returned in skipApps. This is used
for incrementals builds in AL-Go for GitHub.

Remove some excessive logging

If Import-TestToolkitToBcContainer was running with useCompilerFolder,
then it would require a container to be present at that time.

Make Run-AlPipeline do just-in-time creation of CompilerFolder and
Containers - meaning that if you are using useCompilerFolder and do not
have doNotPublishApps set - it will create a compilerFolder and compile
all apps using that, and afterwards - it will create the container,
publish the apps and run the tests.

Also...
Fixes microsoft/AL-Go#1338

---------

Co-authored-by: freddydk <[email protected]>

Author: freddydk

.github/workflows/CI.yaml

@@ -119,7 +119,7 @@ jobs:
           }
 
   Linux:
-    runs-on: [ ubuntu-latest ]
+    runs-on: [ ubuntu-24.04 ]
     defaults:
       run:
         shell: pwsh

.github/workflows/RunTests.yaml

@@ -131,7 +131,7 @@ jobs:
           }
 
   Linux:
-    runs-on: [ ubuntu-latest ]
+    runs-on: [ ubuntu-24.04 ]
     needs: [ AnalyzeTests ]
     if: needs.AnalyzeTests.outputs.linuxtests != '[]'
     strategy:

AppHandling/Run-AlPipeline.ps1

@@ -101,8 +101,8 @@ function Sort-AppFilesByDependencies {
             }
             else {
                 if (-not ($script:unresolvedDependencies | Where-Object { $_ } | Where-Object { "$(if ($_.PSObject.Properties.name -eq 'AppId') { $_.AppId } else { $_.Id })" -eq $dependencyAppId })) {
-                    $appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version)).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
-                    if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972') {
+                    $appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
+                    if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972' -and $dependencyAppId -ne 'f3552374-a1f2-4356-848e-196002525837') {
                         Write-Warning "Dependency $($dependencyAppId):$appFileName not found"
                     }
                     $script:unresolvedDependencies += @($dependency)

AppHandling/Sort-AppFilesByDependencies.ps1

@@ -11,6 +11,10 @@
   If specified, this reference parameter will contain unresolved dependencies after sorting
  .Parameter knownApps
   If specified, this reference parameter will contain all known appids
+ .Parameter skippedApps
+  If specified, this reference parameter will contain all skipped appids
+ .Parameter selectSubordinates
+  If specified, this is the list of appFolders to include (together with subordinates - i.e. appFolders depending on these appFolders)
  .Example
   $folders = Sort-AppFoldersByDependencies -appFolders @($folder1, $folder2)
 #>
@@ -23,7 +27,10 @@ function Sort-AppFoldersByDependencies {
         [Parameter(Mandatory=$false)]
         [ref] $unknownDependencies,
         [Parameter(Mandatory=$false)]
-        [ref] $knownApps
+        [ref] $knownApps,
+        [Parameter(Mandatory=$false)]
+        [ref] $skippedApps,
+        [string[]] $selectSubordinates = @()
     )
 
 $telemetryScope = InitTelemetryScope -name $MyInvocation.InvocationName -parameterValues $PSBoundParameters -includeParameters @()
@@ -40,6 +47,7 @@ try {
     # Read all app.json objects, populate $apps
     $apps = $()
     $folders = @{}
+    $script:includeAppIds = @()
     $appFolders | ForEach-Object {
         $appFolder = "$baseFolder$_"
         $appJsonFile = Join-Path $appFolder "app.json"
@@ -68,14 +76,20 @@ try {
                 $appJson | Add-Member -Name "dependencies" -Type NoteProperty -Value @()
             }
             if ($appJson.psobject.Members | Where-Object name -eq "application") {
-                if ($appJson.Id -ne "63ca2fa4-4f03-4f2b-a480-172fef340d3f") {
+                if ($appJson.Id -ne "63ca2fa4-4f03-4f2b-a480-172fef340d3f" -and $appJson.Id -ne "f3552374-a1f2-4356-848e-196002525837" -and $appJson.Id -ne "437dbf0e-84ff-417a-965d-ed2bb9650972") {
                     $appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "437dbf0e-84ff-417a-965d-ed2bb9650972"; "publisher" = "Microsoft"; "name" = "Base Application"; "version" = $appJson.application }) )
                     $appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "63ca2fa4-4f03-4f2b-a480-172fef340d3f"; "publisher" = "Microsoft"; "name" = "System Application"; "version" = $appJson.application }) )
+                    if ([System.Version]$appJson.application -ge [System.Version]"24.0.0.0") {
+                        $appJson.dependencies += @( New-Object psobject -Property ([ordered]@{ "appId" = "f3552374-a1f2-4356-848e-196002525837"; "publisher" = "Microsoft"; "name" = "Business Foundation"; "version" = $appJson.application }) )
+                    }
                 }
             }
 
             $folders += @{ "$($appJson.Id):$($appJson.Version)" = $appFolder }
             $apps += @($appJson)
+            if ($selectSubordinates -contains $_) {
+                $script:includeAppIds += @($appJson.Id)
+            }
         }
     }
 
@@ -84,50 +98,72 @@ try {
     $script:unresolvedDependencies = $()
 
     function AddAnApp { Param($anApp) 
+        $includeThis = $false
         $alreadyAdded = $script:sortedApps | Where-Object { $_.Id -eq $anApp.Id }
         if (-not ($alreadyAdded)) {
-            AddDependencies -anApp $anApp
+            if (AddDependencies -anApp $anApp) {
+                if ($script:includeAppIds -notcontains $anApp.Id) { 
+                    $script:includeAppIds += @($anApp.Id)
+                }
+                $includeThis = $true
+            }
             $script:sortedApps += $anApp
         }
+        return $includeThis
     }
 
     function AddDependency { Param($dependency)
         $dependencyAppId = "$(if ($dependency.PSObject.Properties.name -eq 'AppId') { $dependency.AppId } else { $dependency.Id })"
+        $includeThis = $script:includeAppIds -contains $dependencyAppId
         $dependentApp = $apps | Where-Object { $_.Id -eq $dependencyAppId } | Sort-Object -Property @{ "Expression" = "[System.Version]Version" }
         if ($dependentApp) {
             if ($dependentApp -is [Array]) {
                 Write-Host -ForegroundColor Yellow "AppFiles contains multiple versions of the app with AppId $dependencyAppId"
                 $dependentApp = $dependentApp | Select-Object -Last 1
             }
-            AddAnApp -AnApp $dependentApp
+            if (AddAnApp -AnApp $dependentApp) {
+                $includeThis = $true
+            }
         }
         else {
             if (-not ($script:unresolvedDependencies | Where-Object { $_ } | Where-Object { "$(if ($_.PSObject.Properties.name -eq 'AppId') { $_.AppId } else { $_.Id })" -eq $dependencyAppId })) {
-                $appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version)).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
-                if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972') {
+                $appFileName = "$($dependency.publisher)_$($dependency.name)_$($dependency.version).app".Split([System.IO.Path]::GetInvalidFileNameChars()) -join ''
+                if ($dependencyAppid -ne '63ca2fa4-4f03-4f2b-a480-172fef340d3f' -and $dependencyAppId -ne '437dbf0e-84ff-417a-965d-ed2bb9650972' -and $dependencyAppId -ne 'f3552374-a1f2-4356-848e-196002525837') {
                     Write-Warning "Dependency $($dependencyAppId):$appFileName not found"
                 }
                 $script:unresolvedDependencies += @($dependency)
             }
         }
+        return $includeThis
     }
 
     function AddDependencies { Param($anApp)
+        $includeThis = $false
         if ($anApp) {
             if ($anApp.psobject.Members | Where-Object name -eq "dependencies") {
                 if ($anApp.Dependencies) {
-                    $anApp.Dependencies | ForEach-Object { AddDependency -Dependency $_ }
+                    $anApp.Dependencies | ForEach-Object {
+                        if (AddDependency -Dependency $_) {
+                            $includeThis = $true
+                        }
+                    }
                 }
             }
         }
+        return $includeThis
     }
 
-    $apps | Where-Object { $_.Name -eq "Application" } | ForEach-Object { AddAnApp -anApp $_ }
-    $apps | ForEach-Object { AddAnApp -AnApp $_ }
+    $apps | Where-Object { $_.Name -eq "Application" } | ForEach-Object { AddAnApp -anApp $_ | Out-Null }
+    $apps | ForEach-Object { AddAnApp -AnApp $_ | Out-Null }
 
     $script:sortedApps | ForEach-Object {
         ($folders["$($_.id):$($_.version)"]).SubString($baseFolder.Length)
     }
+    if ($skippedApps -and $selectSubordinates) {
+        $skippedApps.value = $script:sortedApps | Where-Object { $script:includeAppIds -notcontains $_.id } | ForEach-Object {
+            ($folders["$($_.id):$($_.version)"]).SubString($baseFolder.Length)
+        }
+    }
     if ($knownApps) {
         $knownApps.value += @($script:sortedApps | ForEach-Object {
             $_.Id

AppHandling/Sort-AppFoldersByDependencies.ps1

@@ -1055,7 +1055,7 @@ function GetAppInfo {
         }
         Set-Location (Split-Path $cacheAppInfoPath -parent)
     }
-    Write-GroupStart -Message "Getting .app info $cacheAppInfoPath"
+    Write-Host "Getting .app info $cacheAppInfoPath"
     $binPath = Join-Path $compilerFolder 'compiler/extension/bin'
     $alToolDll = ''
     if ($isLinux) {
@@ -1107,11 +1107,9 @@ function GetAppInfo {
     $package = $null
     try {
         foreach($path in $appFiles) {
-            Write-Host -NoNewline "- $([System.IO.Path]::GetFileName($path))"
             $relativePath = Resolve-Path -Path $path -Relative
             if ($appInfoCache -and $appInfoCache.PSObject.Properties.Name -eq $relativePath) {
                 $appInfo = $appInfoCache."$relativePath"
-                Write-Host " (cached)"
             }
             else {
                 if ($alToolExists) {
@@ -1130,7 +1128,6 @@ function GetAppInfo {
                         "propagateDependencies" = ($manifest.PSObject.Properties.Name -eq 'PropagateDependencies') -and $manifest.PropagateDependencies
                         "dependencies"          = @(if($manifest.PSObject.Properties.Name -eq 'dependencies'){$manifest.dependencies | ForEach-Object { if ($_.PSObject.Properties.Name -eq 'id') { $id = $_.id } else { $id = $_.AppId }; @{ "id" = $id; "name" = $_.name; "publisher" = $_.publisher; "version" = $_.version }}})
                     }
-                    Write-Host " (succeeded using altool)"
                 }
                 else {
                     if (!$assembliesAdded) {
@@ -1158,7 +1155,6 @@ function GetAppInfo {
                         "propagateDependencies" = $manifest.PropagateDependencies
                     }
                     $packageStream.Close()
-                    Write-Host " (succeeded using codeanalysis)"
                 }
                 if ($cacheAppInfoPath) {
                     $appInfoCache | Add-Member -MemberType NoteProperty -Name $relativePath -Value $appInfo
@@ -1183,7 +1179,6 @@ function GetAppInfo {
         }
     }
     catch [System.Reflection.ReflectionTypeLoadException] {
-        Write-Host " (failed)"
         if ($_.Exception.LoaderExceptions) {
             $_.Exception.LoaderExceptions | Select-Object -Property Message | Select-Object -Unique | ForEach-Object {
                 Write-Host "LoaderException: $($_.Message)"
@@ -1200,7 +1195,6 @@ function GetAppInfo {
         }
         Pop-Location
     }
-    Write-GroupEnd
 }
 
 function GetLatestAlLanguageExtensionVersionAndUrl {
@@ -1635,4 +1629,12 @@ function GetHostOs {
         }
     }
     return $hostOs
+}
+
+function Write-PSCallStack {
+    Param(
+        [string] $message
+    )
+    Write-Host "PS CallStack $message :"
+    Get-PSCallStack | ForEach-Object { Write-Host "- $($_.FunctionName) ($([System.IO.Path]::GetFileName($_.ScriptName)) Line $($_.ScriptLineNumber))" }
 }

HelperFunctions.ps1

@@ -39,6 +39,8 @@
   Only Test Runner and Test Framework are/will be available in the online Business Central environment
  .Parameter environment
   Environment in which you want to import test toolkit.
+ .Parameter appSymbolsFolder
+  Folder where the app symbols should be stored when using compilerfolder
  .Example
   Import-TestToolkitToBcContainer -containerName test2
   .Example
@@ -68,8 +70,8 @@ function Import-TestToolkitToBcContainer {
         [switch] $useDevEndpoint,
         [hashtable] $replaceDependencies = $null,
         [Hashtable] $bcAuthContext,
-        [string] $environment
-
+        [string] $environment,
+        [string] $appSymbolsFolder
     )
 
 $telemetryScope = InitTelemetryScope `
@@ -121,6 +123,12 @@ try {
         }
         Write-Host -ForegroundColor Green "TestToolkit successfully published"
     }
+    elseif ($compilerFolder) {
+        $appFiles = GetTestToolkitApps -compilerFolder $compilerFolder -includeTestRunnerOnly:$includeTestRunnerOnly -includeTestFrameworkOnly:$includeTestFrameworkOnly -includeTestLibrariesOnly:$includeTestLibrariesOnly -includePerformanceToolkit:$includePerformanceToolkit
+        $appFiles | ForEach-Object {
+            Copy-Item -Path $_ -Destination $appSymbolsFolder -Force
+        }
+    }
     else {
         $inspect = docker inspect $containerName | ConvertFrom-Json
         if ($inspect.Config.Labels.psobject.Properties.Match('maintainer').Count -eq 0 -or $inspect.Config.Labels.maintainer -ne "Dynamics SMB") {

ObjectHandling/Import-TestToolkitToNavContainer.ps1

@@ -5,6 +5,11 @@ Issue 1303 from AL-Go repository - renew federated token when access token needs
 Issue 3590 EarliestMatching select option added to Download-BcNuGetPackageToFolder
 Regression when from PR 3760 - Push-BcNuGetPackage doesn't work on PowerSHell 5
 Issue 3772 Get-BcNuGetPackageId name length limit
+Two new parameter in Sort-AppFoldersByDependencies - selectSubordinates and SkippedApps. If specified, then only the appFolders specified in the first array plus the apps depending on these recursively will be sorted. The remaining folder not included in the sorted appfolders list will be returned in skippedApps.
+Remove some excessive logging
+If Import-TestToolkitToBcContainer was running with useCompilerFolder, then it would require a container to be present at that time.
+Make Run-AlPipeline do just-in-time creation of CompilerFolder and Containers
+Issue 1338 from AL-Go repository: Multi-Project Repository Test App Dependencies Not Resolved
 
 6.0.29
 Issue 3591 When using Publish-NAVApp to publish an app, which fails compilation in the service, the command might hang forever - the fix for this is a temporary hack put in place for the versions which doesn't work.