Misadventures with [Serializable] During Integration Testing

[MarkNickeson]

Background

I've built an Orleans backend for a webapp and recently began end-to-end integration testing. I use xUnit with fixture collections to setup the various tiers and VS 2022 Preview 4 Test Manager to run/debug tests. Building with Orleans 3.5 and .NET 6 RC1

Symptoms

I initially created 10 smoke tests and eventually had 10/10 successful. However, when I ran the test block again it was 9/10. And after repeated trials it was about 50% of time 10/10, 40% 9/10 and 10% 8/10. Moreover: executing the test block in debug mode would never fail.

Diagnosis

Despite being unable to reproduce the failure while debugging I eventually found the error in the Orleans log: Warning 101029 Orleans.Networking: Unexpected error serializing message. This ultimately pointed me to the fact that BinaryFormatter is deprecated in .NET 5. I had [Serializable] decorating 1 type and only due to a long ago experiment so it was an easy fix.

BUT

I do not understand why the error only manifests periodically. Surely the binary formatter check / failure should happen on every call, particularly with a deterministic test setup. As well, the error did not propagate to the caller as an exception, rather it resulted in a 30 second method invocation timeout.

Questions / Guidance / Experience Sought

1. Is there a way to configure Orleans 3.5 codegen (Microsoft.Orleans.CodeGenerator.MSBuild) to flag [Serializable] with warnings at build time?
2. Is there a logical explanation why the serialization failure does not happen every invocation assuming deterministic setup?

I appreciate any Orleans wisdom that people are willing to share!
Thank you!