[Java.Interop.Tools.Expressions] Add Java.Interop.Tools.Expressions

Fixes: dotnet#616

Context: dotnet#14
Context: ff4053c
Context: da5d1b8
Context: 4787e01
Context: 41ba348

Remember `jnimarshalmethod-gen` (176240d)?  And it's crazy idea to
use the `System.Linq.Expressions`-based custom marshaling
infrastructure (ff4053c, da5d1b8) to generate JNI marshal methods
at build/packaging time.  And how we had to back burner it because
it depended upon `System.Reflection.Emit` being able to write
assemblies to disk, which is a feature that never made it to
.NET Core, and is still lacking as of .NET 7?

Add `src/Java.Interop.Tools.Expressions`, which contains code which
uses Mono.Cecil to compile `Expression<T>` expressions to IL.

Then update `jnimarshalmethod-gen` to use it!

~~ Usage ~~

	% dotnet bin/Debug-net7.0/jnimarshalmethod-gen.dll \
	  bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll \
	  -v -v --keeptemp \
	  --jvm /Library/Java/JavaVirtualMachines/microsoft-11.jdk/Contents/Home/lib/jli/libjli.dylib \
	  -o _x \
	  -L bin/TestDebug-net7.0 \
	  -L /usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.0

First param is assembly to process; `Java.Interop.Export-Tests.dll`
is handy because that's what the `run-test-jnimarshal` target in
`Makefile` processed.

  * `-v -v` is *really* verbose output

  * `--keeptemp` is keep temporary files, in this case
    `_x/Java.Interop.Export-Tests.dll.cecil`.

  * `--jvm PATH` is the path to the JVM library to load+use.

  * `-o DIR` is where to place output files; this will create
    `_x/Java.Interop.Export-Tests.dll`.

  * `-L DIR` adds `DIR` to library resolution paths; this adds
    `bin/TestDebug/net7.0` (dependencies of
    `Java.Interop.Export-Tests.dll`) and
    `Microsoft.NETCore.App/7.0.0-rc.1.22422.12` (net7 libs).

By default the directories containing input assemblies and the
directory containing `System.Private.CoreLib.dll` are part of the
default `-L` list.

When running in-tree, e.g. AzDO pipeline execution, `--jvm PATH`
will attempt to read the path in `bin/Build*/JdkInfo.props`
a'la `TestJVM` (002dea4).  This allows an in-place update in
`core-tests.yaml` which does:

	dotnet bin/Debug-net7.0/jnimarshalmethod-gen.dll \
	  bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll \
	  -v -v --keeptemp -o bin/TestDebug-net7.0

~~ Using `jnimarshalmethod-gen` output ~~

What does `jnimarshalmethod-gen` *do*?

	% ikdasm bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll > beg.il
	% ikdasm _x/Java.Interop.Export-Tests.dll > end.il
	% git diff --no-index beg.il end.il
	# https://gist.github.com/jonpryor/b8233444f2e51043732bea922f6afc81

is a ~1KB diff which shows, paraphrasing greatly:

	public partial class ExportTest {
	    partial class '__<$>_jni_marshal_methods' {
	        static IntPtr funcIJavaObject (IntPtr jnienv, IntPtr this) => …
	        // …
	        [JniAddNativeMethodRegistration]
	        static void __RegisterNativeMembers (JniNativeMethodRegistrationArguments args) => …
	    }
	}
	internal delegate long _JniMarshal_PP_L (IntPtr jnienv, IntPtr self);
	// …

wherein `ExportTest._<$>_jni_marshal_methods` and the `_JniMarshal*`
delegate types are added to the assembly.

This also unblocks the desire stated in 4787e01:

> For `Java.Base`, @jonpryor wants to support the custom marshaling
> infrastructure introduced in 77a6bf8.  This would allow types to
> participate in JNI marshal method ("connector method") generation
> *at runtime*, allowing specialization based on the current set of
> types and assemblies.

What can we do with this `jnimarshalmethod-gen` output?  Use it!

First, make sure the tests work:

	% dotnet test --logger "console;verbosity=detailed" bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll
	…
	Passed!  - Failed:     0, Passed:    17, Skipped:     0, Total:    17, Duration: 103 ms - Java.Interop.Export-Tests.dll (net7.0)

Then update/replace the unit test assembly with
`jnimarshalmethod-gen` output:

	% \cp _x/Java.Interop.Export-Tests.dll bin/TestDebug-net7.0
	% dotnet test --logger "console;verbosity=detailed" bin/TestDebug-net7.0/Java.Interop.Export-Tests.dll
	…
	Total tests: 17
	     Passed: 17

`core-tests.yaml` has been updated to do this.

~~ One-Off Tests ~~

One-off tests: ensure that the generated assembly can be decompiled:

	% ikdasm  bin/TestDebug-net7.0/Java.Interop.Tools.Expressions-Tests-ExpressionAssemblyBuilderTests.dll
	% monodis bin/TestDebug-net7.0/Java.Interop.Tools.Expressions-Tests-ExpressionAssemblyBuilderTests.dll

	% ikdasm  _x/Java.Interop.Export-Tests.dll
	% monodis _x/Java.Interop.Export-Tests.dll
	# which currently fails :-()

Re-enable most of `Java.Interop.Export-Tests.dll` for .NET 7;
see 41ba348, which disabled those tests.

To verify the generated IL, use the [dotnet-ilverify][0] tool:

	dotnet tool install --global dotnet-ilverify

Usage of which is "weird":

	$HOME/.dotnet/tools/ilverify _x/Java.Interop.Export-Tests.dll \
	  --tokens --system-module System.Private.CoreLib \
	  -r 'bin/TestDebug-net7.0/*.dll' \
	  -r '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.0/*.dll'
	All Classes and Methods in /Volumes/Xamarin-Work/src/xamarin/Java.Interop/_x/Java.Interop.Export-Tests.dll Verified.
	# no errors!

where:

  * `--tokens`: Include metadata tokens in error messages.
  * `--system-module NAME`: set the "System module name".  Defaults
    to `mscorlib`, which is wrong for .NET 5+, so this must be set to
    `System.Private.CoreLib` (no `.dll` suffix!).
  * `-r FILE-GLOB`: Where to resolve assembly references for the
    input assembly.  Fortunately file globs are supported…

~~ Removing `System.Private.CoreLib` ~~

`System.Private.CoreLib.dll` is *private*; it's not part of the
public assembly surface area, so you can't use
`csc -r:System.Private.CoreLib …` and expect it to work.  This makes
things interesting because *at runtime* everything "important" is in
`System.Private.CoreLib.dll`, like `System.Object`.

Specifically, if we do the "obvious" thing and do:

	newTypeDefinition.BaseType = assemblyDefinition.MainModule
	    .ImportReference (typeof (object));

you're gonna have a bad type, because the resulting IL for
`newTypeDefinition` will have a base class of
`[System.Private.CoreLib]System.Object`, which isn't usable.

Fix this by:

 1. Writing the assembly to a `Stream`.
 2. Reading the `Stream` from (1)
 3. Fixing all member references and assembly references so that
    `System.Private.CoreLib` is not referenced.

If `jnimarshalmethod-gen.dll --keeptemp` is used, then a `.cecil`
file is created with the contents of (1).

Additionally, and unexpectedly -- [jbevain/cecil#895][1] -- Mono.Cecil
adds a reference to the assembly being modified.  Remove the declaring
assembly from `AssemblyReferences`.

[0]: https://www.nuget.org/packages/dotnet-ilverify
[1]: jbevain/cecil#895

Author: jonpryor

Diff for: Java.Interop.sln

@@ -109,6 +109,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Base", "src\Java.Base\
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Base-Tests", "tests\Java.Base-Tests\Java.Base-Tests.csproj", "{CB05E11B-B96F-4179-A4E9-5D6BDE29A8FC}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Interop.Tools.Expressions", "src\Java.Interop.Tools.Expressions\Java.Interop.Tools.Expressions.csproj", "{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Interop.Tools.Expressions-Tests", "tests\Java.Interop.Tools.Expressions-Tests\Java.Interop.Tools.Expressions-Tests.csproj", "{211BAA88-66B1-41B2-88B2-530DBD8DF702}"
+EndProject
 Global
 	GlobalSection(SharedMSBuildProjectFiles) = preSolution
 		src\Java.Interop.NamingCustomAttributes\Java.Interop.NamingCustomAttributes.projitems*{58b564a1-570d-4da2-b02d-25bddb1a9f4f}*SharedItemsImports = 5
@@ -308,6 +312,14 @@ Global
 		{CB05E11B-B96F-4179-A4E9-5D6BDE29A8FC}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{CB05E11B-B96F-4179-A4E9-5D6BDE29A8FC}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{CB05E11B-B96F-4179-A4E9-5D6BDE29A8FC}.Release|Any CPU.Build.0 = Release|Any CPU
+		{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A}.Release|Any CPU.Build.0 = Release|Any CPU
+		{211BAA88-66B1-41B2-88B2-530DBD8DF702}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{211BAA88-66B1-41B2-88B2-530DBD8DF702}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{211BAA88-66B1-41B2-88B2-530DBD8DF702}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{211BAA88-66B1-41B2-88B2-530DBD8DF702}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -360,6 +372,8 @@ Global
 		{11942DE9-AEC2-4B95-87AB-CA707C37643D} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
 		{30DCECA5-16FD-4FD0-883C-E5E83B11565D} = {0998E45F-8BCE-4791-A944-962CD54E2D80}
 		{CB05E11B-B96F-4179-A4E9-5D6BDE29A8FC} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
+		{1A0262FE-3CDB-4AF2-AAD8-65C59524FE8A} = {0998E45F-8BCE-4791-A944-962CD54E2D80}
+		{211BAA88-66B1-41B2-88B2-530DBD8DF702} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {29204E0C-382A-49A0-A814-AD7FBF9774A5}

Diff for: build-tools/automation/templates/core-tests.yaml

@@ -105,13 +105,31 @@ steps:
 
 - task: DotNetCoreCLI@2
   displayName: 'Tests: Java.Interop.Export'
-  condition: eq('${{ parameters.runNativeTests }}', 'true')
+  condition: or(eq('${{ parameters.runNativeDotnetTests }}', 'true'), eq('${{ parameters.runNativeTests }}', 'true'))
   inputs:
     command: test
     testRunTitle: Java.Interop.Export (${{ parameters.platformName }})
     arguments: bin/Test$(Build.Configuration)$(NetCoreTargetFrameworkPathSuffix)/Java.Interop.Export-Tests.dll
   continueOnError: true
 
+- task: DotNetCoreCLI@2
+  displayName: 'jnimarshalmethod-gen Java.Interop.Export-Tests.dll'
+  condition: eq('${{ parameters.runNativeDotnetTests }}', 'true')
+  inputs:
+    command: custom
+    custom: bin/$(Build.Configuration)$(NetCoreTargetFrameworkPathSuffix)/jnimarshalmethod-gen.dll
+    arguments: bin/Test$(Build.Configuration)$(NetCoreTargetFrameworkPathSuffix)/Java.Interop.Export-Tests.dll -v -v --keeptemp -o bin/Test$(Build.Configuration)$(NetCoreTargetFrameworkPathSuffix)
+  continueOnError: true
+
+- task: DotNetCoreCLI@2
+  displayName: 'Tests: Java.Interop.Export w/ jnimarshalmethod-gen!'
+  condition: or(eq('${{ parameters.runNativeDotnetTests }}', 'true'), eq('${{ parameters.runNativeTests }}', 'true'))
+  inputs:
+    command: test
+    testRunTitle: Java.Interop.Export (jnimarshalmethod-gen + ${{ parameters.platformName }})
+    arguments: bin/Test$(Build.Configuration)$(NetCoreTargetFrameworkPathSuffix)/Java.Interop.Export-Tests.dll
+  continueOnError: true
+
 - task: DotNetCoreCLI@2
   displayName: 'Tests: Java.Interop-Performance-net472'
   condition: eq('${{ parameters.runNativeTests }}', 'true')

Diff for: src/Java.Base-ref.cs

@@ -6408,7 +6408,7 @@ public partial class AccessibleObject : Java.Lang.Object, Java.Interop.IJavaPeer
     {
         protected AccessibleObject() { }
         protected AccessibleObject(ref Java.Interop.JniObjectReference reference, Java.Interop.JniObjectReferenceOptions options) { }
-        public virtual bool Accessible { get { throw null; } set { } }
+        public virtual bool Accessible { [System.ObsoleteAttribute("deprecated")] get { throw null; } set { } }
         [System.ComponentModel.EditorBrowsableAttribute(1)]
         [System.Diagnostics.DebuggerBrowsableAttribute(0)]
         public override Java.Interop.JniPeerMembers JniPeerMembers { get { throw null; } }

Diff for: src/Java.Interop.Export/Java.Interop.Export.csproj

@@ -2,7 +2,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
   <PropertyGroup>
     <TargetFrameworks>netstandard2.0;$(DotNetTargetFramework)</TargetFrameworks>
-    <LangVersion>8.0</LangVersion>
+    <LangVersion>9.0</LangVersion>
     <ProjectGuid>{B501D075-6183-4E1D-92C9-F7B5002475B1}</ProjectGuid>
     <Nullable>enable</Nullable>
     <SignAssembly>true</SignAssembly>
@@ -23,4 +23,4 @@
   <ItemGroup>
     <PackageReference Include="Microsoft.SourceLink.GitHub" PrivateAssets="All" />
   </ItemGroup>
-</Project>
+</Project>

Diff for: src/Java.Interop.Export/Java.Interop/MarshalMemberBuilder.cs

@@ -84,20 +84,6 @@ public string GetJniMethodSignature (JavaCallableAttribute export, MethodInfo me
 			return export.Signature = GetJniMethodSignature (method);
 		}
 
-		string GetTypeSignature (ParameterInfo p)
-		{
-			var info        = Runtime.TypeManager.GetTypeSignature (p.ParameterType);
-			if (info.IsValid)
-				return info.QualifiedReference;
-
-			var marshaler   = GetParameterMarshaler (p);
-			info            = Runtime.TypeManager.GetTypeSignature (marshaler.MarshalType);
-			if (info.IsValid)
-				return info.QualifiedReference;
-
-			throw new NotSupportedException ("Don't know how to determine JNI signature for parameter type: " + p.ParameterType.FullName + ".");
-		}
-
 		Delegate CreateJniMethodMarshaler (MethodInfo method, JavaCallableAttribute? export, Type? type)
 		{
 			var e = CreateMarshalToManagedExpression (method, export, type);
@@ -242,6 +228,7 @@ public LambdaExpression CreateMarshalToManagedExpression (MethodInfo method, Jav
 				: Expression.Lambda (marshalerType, body, bodyParams);
 		}
 
+		// Keep in sync with ExpressionAssemblyBuilder.GetMarshalMethodDelegateType()
 		static Type? GetMarshalerType (Type? returnType, List<Type> funcTypeParams, Type? declaringType)
 		{
 			// Too many parameters; does a `_JniMarshal_*` type exist in the type's declaring assembly?
@@ -277,6 +264,7 @@ public LambdaExpression CreateMarshalToManagedExpression (MethodInfo method, Jav
 		static AssemblyBuilder? assemblyBuilder;
 		static ModuleBuilder?   moduleBuilder;
 		static Type[]?          DelegateCtorSignature;
+		static Dictionary<string, Type>? marshalDelegateTypes;
 
 		static Type? CreateMarshalDelegateType (string name, Type? returnType, List<Type> funcTypeParams)
 		{
@@ -290,6 +278,10 @@ public LambdaExpression CreateMarshalToManagedExpression (MethodInfo method, Jav
 						typeof (object),
 						typeof (IntPtr)
 					};
+					marshalDelegateTypes = new ();
+				}
+				if (marshalDelegateTypes!.TryGetValue (name, out var type)) {
+					return type;
 				}
 				funcTypeParams.Insert (0, typeof (IntPtr));
 				funcTypeParams.Insert (0, typeof (IntPtr));
@@ -307,7 +299,9 @@ public LambdaExpression CreateMarshalToManagedExpression (MethodInfo method, Jav
 					.SetImplementationFlags (ImplAttributes);
 				typeBuilder.DefineMethod ("Invoke", InvokeAttributes, returnType, funcTypeParams.ToArray ())
 					.SetImplementationFlags (ImplAttributes);
-				return typeBuilder.CreateTypeInfo ();
+				var marshalDelType = typeBuilder.CreateTypeInfo ();
+				marshalDelegateTypes.Add (name, marshalDelType);
+				return marshalDelType;
 			}
 		}
 #endif  // NET

Diff for: src/Java.Interop.Tools.Expressions/Java.Interop.Tools.Expressions.csproj

@@ -0,0 +1,27 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>$(DotNetTargetFramework)</TargetFramework>
+    <ImplicitUsings>enable</ImplicitUsings>
+    <Nullable>enable</Nullable>
+  </PropertyGroup>
+
+  <Import Project="..\..\TargetFrameworkDependentValues.props" />
+
+  <PropertyGroup>
+    <OutputPath>$(UtilityOutputFullPath)</OutputPath>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.SourceLink.GitHub" PrivateAssets="All" />
+  </ItemGroup>
+
+  <Import Project="..\..\build-tools\scripts\cecil.projitems" />
+
+  <ItemGroup>
+    <ProjectReference Include="..\..\src\Java.Interop\Java.Interop.csproj" />
+    <ProjectReference Include="..\..\src\Java.Interop.Tools.Cecil\Java.Interop.Tools.Cecil.csproj" />
+    <ProjectReference Include="..\..\src\Java.Interop.Tools.Diagnostics\Java.Interop.Tools.Diagnostics.csproj" />
+  </ItemGroup>
+
+</Project>