Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

AlexRadch · 2025-01-10T23:17:20Z

This commit introduces the System.Text.Unicode.Utf8 type to the Microsoft.Bcl.Memory library.

It includes type forwarding for Utf8 in Microsoft.Bcl.Memory.Forwards.cs, updates the documentation in PACKAGE.md to include Utf8 functionality, and adds corresponding test cases in Microsoft.Bcl.Memory.Tests.csproj.

The documentation now emphasizes Utf8 alongside Index, Range, and Base64Url, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

This commit introduces the `System.Text.Unicode.Utf8` type to the `Microsoft.Bcl.Memory` library. It includes type forwarding for `Utf8` in `Microsoft.Bcl.Memory.Forwards.cs`, updates the documentation in `PACKAGE.md` to include `Utf8` functionality, and adds corresponding test cases in `Microsoft.Bcl.Memory.Tests.csproj`. The documentation now emphasizes `Utf8` alongside `Index`, `Range`, and `Base64Url`, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

stephentoub · 2025-01-11T00:47:49Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

Updated `PackageDescription` to include "Utf8" support. Added new `ItemGroup` for conditional compilation of UTF-8 and Unicode handling files for non-net8.0 frameworks. Modified visibility and implementations in `Ascii.Utility.Helpers.cs`, `Utf8.cs`, and `Utf8Utility` based on `MICROSOFT_BCL_MEMORY` define.

AlexRadch · 2025-01-11T12:53:28Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

I added Utf8 APIs to Microsoft.Bcl.Memory library. I haven't added tests yet. I'll add them too.

Updated `Microsoft.Bcl.Memory.Tests.csproj` to include `UnicodeUtility.cs` and removed .NET 8.0 targeting condition. Modified `Utf8Tests.cs` by adjusting using directives and enhancing the `DecodeHex` method with conditional compilation for .NET 5.0+.

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj

src/libraries/Microsoft.Bcl.Memory/src/PACKAGE.md

AlexRadch · 2025-01-11T15:39:24Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

@stephentoub I added Utf8 APIs to Microsoft.Bcl.Memory library for older .NET platforms. I've also added tests for older .NET platforms.

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

teo-tsirpanis · 2025-01-11T18:20:50Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

@@ -9,7 +9,10 @@

 namespace System.Text.Unicode
 {
-#if SYSTEM_PRIVATE_CORELIB
+#if SYSTEM_PRIVATE_CORELIB || MICROSOFT_BCL_MEMORY


This file is being referenced only by System.Private.CoreLib and Microsoft.BCL.Memory, so the class can always be public.

I'm afraid to make such changes and let someone else make them as a separate PR.

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs

teo-tsirpanis · 2025-01-11T18:45:40Z

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs

+#if !MICROSOFT_BCL_MEMORY
+        public
+#else
+        private
+#endif
+            static byte[] ToUtf8(Rune rune)


Suggested change

#if !MICROSOFT_BCL_MEMORY

public

#else

private

#endif

static byte[] ToUtf8(Rune rune)

public static byte[] ToUtf8(Rune rune)

Why can't this always be public?

In tests for older versions of .Net where there is no Rune class yet, I create it as an internal class, and the compiler complains that the public API cannot reference internal classes.

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.Forwards.cs

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj

Co-authored-by: Theodore Tsirpanis <[email protected]>

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

teo-tsirpanis · 2025-01-19T15:21:25Z

(removing NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) as the reason it was added has been addressed)

teo-tsirpanis · 2025-02-23T22:26:20Z

@dotnet/area-system-memory could you review this? Thanks.

jkotas · 2025-02-23T22:35:54Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

+        /// the data could not be successfully decoded. This pattern provides convenient automatic U+FFFD substitution of
+        /// invalid sequences while iterating through the loop.
+        /// </remarks>
+        private static OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> source, out uint result, out int bytesConsumed)


Can the project include Rune.cs instead of including a copy of the implementation here?

Done in teo-tsirpanis@79ee05d in order to unblock work that depends on this, after waiting for one week. Anyone with permissions can push my commit on the PR's branch.

Compile errors fixed in teo-tsirpanis@bf6f989.

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

(cherry picked from commit bf6f989)

teo-tsirpanis · 2025-03-10T14:05:01Z

CI failures are unrelated.

jkotas · 2025-03-10T22:16:43Z

I do not have any additional comments. @dotnet/area-system-memory should review and signoff.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 10, 2025

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 10, 2025

stephentoub added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jan 11, 2025

teo-tsirpanis added area-System.Memory and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jan 11, 2025

build-analysis bot mentioned this pull request Jan 11, 2025

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

AlexRadch added 2 commits January 11, 2025 21:49

Add compilation constant for Microsoft BCL Memory

22be776

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

akoeplinger reviewed Jan 11, 2025

View reviewed changes

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj Outdated Show resolved Hide resolved

src/libraries/Microsoft.Bcl.Memory/src/PACKAGE.md Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Jan 11, 2025

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

teo-tsirpanis reviewed Jan 11, 2025

View reviewed changes

build-analysis bot mentioned this pull request Jan 11, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

AlexRadch and others added 5 commits January 12, 2025 08:29

Apply suggestions from code review

79214bd

Co-authored-by: Theodore Tsirpanis <[email protected]>

Update Microsoft.Bcl.Memory for framework compatibility

def40c4

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Space

a1caf07

Remove MICROSOFT_BCL_MEMORY from project constants

695e362

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

Refactor Utf8Tests for .NET compatibility improvements

063359a

teo-tsirpanis mentioned this pull request Jan 12, 2025

[API Proposal]: Extend BlobBuilder so consumers can better control allocations #100418

Open

teo-tsirpanis removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jan 19, 2025

jkotas reviewed Feb 23, 2025

View reviewed changes

Use Rune.DecodeFromUtf8 on all frameworks.

b5992c6

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

build-analysis bot mentioned this pull request Mar 4, 2025

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

Open

Fix compile errors.

90a3613

(cherry picked from commit bf6f989)

build-analysis bot mentioned this pull request Mar 10, 2025

Assertion failed in SafeFileHandle.OverlappedValueTaskSource.IOCallback #99517

Open

akoeplinger requested a review from jkotas March 10, 2025 16:17

jkotas requested review from a team and removed request for jkotas March 10, 2025 22:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

AlexRadch commented Jan 10, 2025

stephentoub commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

teo-tsirpanis Jan 11, 2025

AlexRadch Jan 12, 2025

teo-tsirpanis Jan 11, 2025

AlexRadch Jan 12, 2025

teo-tsirpanis commented Jan 19, 2025

teo-tsirpanis commented Feb 23, 2025

jkotas Feb 23, 2025

teo-tsirpanis Mar 3, 2025

akoeplinger Mar 4, 2025

teo-tsirpanis Mar 8, 2025 •

edited

Loading

akoeplinger Mar 10, 2025

teo-tsirpanis commented Mar 10, 2025

jkotas commented Mar 10, 2025

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Are you sure you want to change the base?

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Conversation

AlexRadch commented Jan 10, 2025

stephentoub commented Jan 11, 2025 • edited Loading

AlexRadch commented Jan 11, 2025 • edited Loading

AlexRadch commented Jan 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis commented Jan 19, 2025

teo-tsirpanis commented Feb 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis Mar 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teo-tsirpanis commented Mar 10, 2025

jkotas commented Mar 10, 2025

stephentoub commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

teo-tsirpanis Mar 8, 2025 •

edited

Loading