[XABT] Create separate assembly preparation tasks. #9637

jpobst · 2024-12-19T19:04:54Z

Break down the CollectAssemblyFilesForArchive task further into individual tasks.

These tasks each handle a separate concern about preparing the assemblies for the apk:

CompressAssemblies - Compresses assemblies using LZ4 compression (independent of ZIP compression).
CreateAssemblyStore - Bundles assemblies into a single .blob file.
WrapAssembliesAsSharedLibraries - Wraps the assemblies as shared libraries so they can go in the lib directory of the APK.

This should allow them to be moved to separate incremental targets in the future. For example, we do not need to compress all assemblies on every build if they haven't changed since the previous build.

dellis1972 · 2025-01-06T18:15:30Z

src/Xamarin.Android.Build.Tasks/Utilities/ITaskItemExtensions.cs

 		{
 			var value = item.GetMetadata (name);

 			if (string.IsNullOrWhiteSpace (value))
 				return defaultValue;

-			return value;
+			return (T?)Convert.ChangeType (value, typeof (T));


This throws https://learn.microsoft.com/en-us/dotnet/api/system.convert.changetype?view=net-9.0. We should handle that case? and return the default.

I think this only throws if the input cannot be converted to the requested type, like say you passed in "tru" and T is bool.

Passing in an invalid input is a bug that should be fixed, and it feels like we should indeed throw an exception here rather than silently swallow the error and return the default.

jonpryor · 2025-01-13T23:22:09Z

Recording what was mentioned on an earlier call:

We need to simplify our build system (and related) to reduce maintenance costs, up to and including removing infrequently/rarely used features.

While we need to continue to support $(EmbedAssembliesIntoApk)=True, that does not meant that we need to continue to support "one file per assembly within the .apk" (86260ed), which is Yet Another Codepath that we need to support for minimal gain. $(EmbedAssembliesIntoApk)=True should instead be an Assembly Store, even for Debug builds.

The appropriate fix for #9455 is not to "make $(EmbedAssembliesIntoApk)=True faster"; it is instead to make Fast Deployment more reliable. If Fast Deployment is only successful 40% of the time, we need more information (adb logcat output, MSBuild .binlog files around dotnet build -t:Install) so that we can fix those reliability issues.

jpobst · 2025-01-14T19:56:30Z

$(EmbedAssembliesIntoApk)=True should instead be an Assembly Store, even for Debug builds.

One caveat with this today is that assembly stores do not contain .pdb files, so you can't debug when using assembly stores. I would assume this is fixable but may require runtime loader changes. 🤷

android/src/Xamarin.Android.Build.Tasks/Tasks/CollectAssemblyFilesForArchive.cs

Lines 118 to 153 in 0475a60

    
           void DoAddAssembliesFromArchCollection (TaskLoggingHelper log, AndroidTargetArch arch, ITaskItem assembly, DSOWrapperGenerator.Config dsoWrapperConfig, PackageFileListBuilder files, bool debug, bool compress, IDictionary<AndroidTargetArch, Dictionary<string, CompressedAssemblyInfo>>? compressedAssembliesInfo, string compressedOutputDir, AssemblyStoreBuilder? storeBuilder) 
        
           { 
        
           	// In the "all assemblies are per-RID" world, assemblies, pdb and config are disguised as shared libraries (that is, 
        
           	// their names end with the .so extension) so that Android allows us to put them in the `lib/{ARCH}` directory. 
        
           	// For this reason, they have to be treated just like other .so files, as far as compression rules are concerned. 
        
           	// Thus, we no longer just store them in the apk but we call the `GetCompressionMethod` method to find out whether 
        
           	// or not we're supposed to compress .so files. 
        
           	var sourcePath = CompressAssembly (assembly, compress, compressedAssembliesInfo, compressedOutputDir); 
        
           	if (UseAssemblyStore) { 
        
           		storeBuilder!.AddAssembly (sourcePath, assembly, includeDebugSymbols: debug); 
        
           		return; 
        
           	} 
        
           	// Add assembly 
        
           	(string assemblyPath, string assemblyDirectory) = GetInArchiveAssemblyPath (assembly); 
        
           	string wrappedSourcePath = DSOWrapperGenerator.WrapIt (Log, dsoWrapperConfig, arch, sourcePath, Path.GetFileName (assemblyPath)); 
        
           	files.AddItem (wrappedSourcePath, assemblyPath); 
        
           	// Try to add config if exists 
        
           	var config = Path.ChangeExtension (assembly.ItemSpec, "dll.config"); 
        
           	AddAssemblyConfigEntry (dsoWrapperConfig, files, arch, assemblyDirectory, config); 
        
           	// Try to add symbols if Debug 
        
           	if (!debug) { 
        
           		return; 
        
           	} 
        
           	string symbols = Path.ChangeExtension (assembly.ItemSpec, "pdb"); 
        
           	if (!File.Exists (symbols)) { 
        
           		return; 
        
           	} 
        
           	string archiveSymbolsPath = assemblyDirectory + MonoAndroidHelper.MakeDiscreteAssembliesEntryName (Path.GetFileName (symbols)); 
        
           	string wrappedSymbolsPath = DSOWrapperGenerator.WrapIt (Log, dsoWrapperConfig, arch, symbols, Path.GetFileName (archiveSymbolsPath)); 
        
           	files.AddItem (wrappedSymbolsPath, archiveSymbolsPath); 
        
           }

We default assembly stores to false if using debug symbols:

android/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets

Lines 328 to 329 in 0475a60

    
           <AndroidUseAssemblyStore Condition=" '$(AndroidUseAssemblyStore)' == '' and ('$(EmbedAssembliesIntoApk)' != 'true' or '$(AndroidIncludeDebugSymbols)' == 'true') ">false</AndroidUseAssemblyStore> 
        
           <AndroidUseAssemblyStore Condition=" '$(AndroidUseAssemblyStore)' == '' ">true</AndroidUseAssemblyStore>

…aries' tasks.

Copilot reviewed 5 out of 8 changed files in this pull request and generated no comments.

Files not reviewed (3)

src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets: Language not supported
src/Xamarin.Android.Build.Tasks/Tasks/CollectAssemblyFilesForArchive.cs: Evaluated as low risk
src/Xamarin.Android.Build.Tasks/Tasks/CompressAssemblies.cs: Evaluated as low risk

Comments suppressed due to low confidence (1)

src/Xamarin.Android.Build.Tasks/Tasks/CreateAssemblyStore.cs:79

The metadata value may not be a valid boolean. Consider using bool.TryParse to ensure the metadata value is a valid boolean.

var should_skip = asm.GetMetadataOrDefault ("AndroidSkipAddToPackage", false);

grendello · 2025-01-16T08:41:01Z

$(EmbedAssembliesIntoApk)=True should instead be an Assembly Store, even for Debug builds.

One caveat with this today is that assembly stores do not contain .pdb files, so you can't debug when using assembly stores. I would assume this is fixable but may require runtime loader changes. 🤷

They support PDB files (and also per assembly .config files). If debugging with assembly stores doesn't work, then perhaps the problem is in Mono not getting them (or not asking for them) when debugging.

jonathanpeppers · 2025-01-16T15:42:18Z

src/Xamarin.Android.Build.Tasks/Tasks/CompressAssemblies.cs

+		var key = CompressedAssemblyInfo.GetKey (ProjectFullPath);
+		Log.LogDebugMessage ($"Retrieving assembly compression info with key '{key}'");
+
+		var compressedAssembliesInfo = BuildEngine4.UnregisterTaskObjectAssemblyLocal<IDictionary<AndroidTargetArch, Dictionary<string, CompressedAssemblyInfo>>> (key, RegisteredTaskObjectLifetime.Build);
+
+		if (compressedAssembliesInfo is null)
+			throw new InvalidOperationException ($"Assembly compression info not found for key '{key}'. Compression will not be performed.");


The old code was like this, but...

What saves this RegisterTaskObject value? If we can avoid it, I wouldn't recommend using RegisterTaskObject here as a project could have:

<TargetFrameworks>net9.0-android;net10.0-android</TargetFrameworks>

The two would overwrite each other with this value for the key. The CompressedAssemblyInfo type will be incompatible between the two assemblies under .NET framework because their versions do not match.

Can we calculate the value within this task and not use RegisterTaskObject?

It's generated here:

android/src/Xamarin.Android.Build.Tasks/Tasks/GenerateCompressedAssembliesNativeSourceFiles.cs

Line 88 in fec3b79

string key = CompressedAssemblyInfo.GetKey (ProjectFullPath);

The problem is that we have to generate the LLVM IR assembly sources way, way before we package the assemblies and the info gathered while generating the native assembly sources is crucial to the correct construction of the assembly store. We allocate buffers of specific size for each of the assemblies, so that decompressed data for each assembly gets a buffer of the correct size. This saves memory. The way it works is that each assembly has an index, allocated while generating the native assembler sources, and that index is used at run time to find the correct buffer.

Maybe the assemblies could be compressed just before generation step for the native assembler sources, but I don't know if it isn't too early.

From what I could tell, the only thing that gets used from this is the "DescriptorIndex":

android/src/Xamarin.Android.Build.Tasks/Utilities/AssemblyCompression.cs

Line 136 in f9f421b

AssemblyData compressedAssembly = new AssemblyData (assembly.ItemSpec, info.DescriptorIndex);

It feels like this could be a piece of metadata added to the @(Item) instead, but I certainly haven't decoded the full picture.

Any change here should likely be done in its own PR.

jpobst · 2025-01-17T01:47:49Z

They support PDB files (and also per assembly .config files). If debugging with assembly stores doesn't work, then perhaps the problem is in Mono not getting them (or not asking for them) when debugging.

It looks like here's the context for why assembly stores are disabled by default when debugging: #6660.

jonathanpeppers · 2025-01-17T21:53:51Z

#6660 seems like it was specific to DebugType=full (not even supported by .NET 6+).

In the original issue, a breakpoint didn't even work. So, we should have a test to catch that if it was broken now.

* main: [XABT] Create separate assembly preparation tasks. (#9637) Add `activity-alias` support (#9654) [CI] Grab 'dotnet-install.sh' script directly from GitHub. (#9699)

jpobst force-pushed the separate-compress branch from 0e05e6a to 1ce7fb1 Compare December 19, 2024 19:19

dellis1972 reviewed Jan 6, 2025

View reviewed changes

[XABT] Create separate compress assemblies task.

dc8613c

jpobst force-pushed the separate-compress branch from 1ce7fb1 to 33cee50 Compare January 14, 2025 21:34

dotnet deleted a comment from azure-pipelines bot Jan 14, 2025

Create separate 'CreateAssemblyStore' and 'WrapAssembliesAsSharedLibr…

ea10b6f

…aries' tasks.

jpobst force-pushed the separate-compress branch from 33cee50 to ea10b6f Compare January 15, 2025 18:45

dotnet deleted a comment from azure-pipelines bot Jan 15, 2025

jpobst changed the title ~~[XABT] Create separate compress assemblies task.~~ [XABT] Create separate assembly preparation tasks. Jan 15, 2025

jpobst marked this pull request as ready for review January 15, 2025 20:35

jpobst requested a review from jonathanpeppers as a code owner January 15, 2025 20:35

jpobst requested review from jonpryor, grendello and Copilot January 15, 2025 20:35

Copilot AI reviewed Jan 15, 2025

View reviewed changes

grendello approved these changes Jan 16, 2025

View reviewed changes

jonathanpeppers reviewed Jan 16, 2025

View reviewed changes

jonathanpeppers approved these changes Jan 17, 2025

View reviewed changes

jpobst merged commit 80bb5d7 into main Jan 22, 2025
56 of 59 checks passed

jpobst deleted the separate-compress branch January 22, 2025 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[XABT] Create separate assembly preparation tasks. #9637

[XABT] Create separate assembly preparation tasks. #9637

jpobst commented Dec 19, 2024 •

edited

Loading

dellis1972 Jan 6, 2025

jpobst Jan 6, 2025 •

edited

Loading

jonpryor commented Jan 13, 2025

jpobst commented Jan 14, 2025

grendello commented Jan 16, 2025

jonathanpeppers Jan 16, 2025

grendello Jan 16, 2025

jpobst Jan 16, 2025

jpobst commented Jan 17, 2025

jonathanpeppers commented Jan 17, 2025

[XABT] Create separate assembly preparation tasks. #9637

[XABT] Create separate assembly preparation tasks. #9637

Conversation

jpobst commented Dec 19, 2024 • edited Loading

dellis1972 Jan 6, 2025

Choose a reason for hiding this comment

jpobst Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

jonpryor commented Jan 13, 2025

jpobst commented Jan 14, 2025

Choose a reason for hiding this comment

grendello commented Jan 16, 2025

jonathanpeppers Jan 16, 2025

Choose a reason for hiding this comment

grendello Jan 16, 2025

Choose a reason for hiding this comment

jpobst Jan 16, 2025

Choose a reason for hiding this comment

jpobst commented Jan 17, 2025

jonathanpeppers commented Jan 17, 2025

jpobst commented Dec 19, 2024 •

edited

Loading

jpobst Jan 6, 2025 •

edited

Loading