Using multiple DataRowsAttributes makes the test non CLS-compliant

[martinsuchan]

The change #1503 and #1646 introduced regression where test methods with multiple DataRowsAttributes are now no longer CLS-compliant.

I discovered this issue in our codebase where we use shared AssemblyInfo for all projects including tests with
[assembly: CLSCompliant(true)]
Together with "Treat warnings as errors" that we use updating MSTest rom 3.0.1 to 3.1.1 effectively breaks our build.
I know this is probably very edge case scenario, CLS complinace in test projects has not much meaning, but in our case I need to manually add CLSCompliant(false) to about dozen test classes to fix it.

Expected behavior

No breaking change when updating MSTest from 3.0.1 to 3.1.1

Actual behavior

Cannot build solution without manually changing dozen+ of our test files.

Simple repro:

AB#1925774

[martinsuchan]

Note, can be reproduced in VS 2022 17.7.4 in net472 and net6.0 test project.

[Evangelink]

Hi @martinsuchan! Thanks for the bug report. I definitely need to add a test case for CLS compliant as you are not the first one reporting an issue with it. We will work on a fix asap :)

[Evangelink]

Hi @martinsuchan!

Sorry for the delay in handling this issue. I am working on a fix and would like a small confirmation that all cases raising an issue are cases where you have only one item declared?

[martinsuchan]

Hi, just a note, you should probably add more tests to the ClsTests class, including with zero, one or more DataRows.
We are using multiple DataRows of items in our tests.
Also the problem I reported was not with tests not passing, but with build warnings. I don't see any place checking if there are no build warning about CLS compliance?
8c9530e

[Evangelink]

Reopening to explore the testing part. Thanks @martinsuchan artin

[Evangelink]

@martinsuchan I have done a couple of manual tests and it seems that there is no strong need for a more complex test. Because the test assets are built, if we regress, we will start to have some warnings triggered and because we have warnings as errors in CI, this would ensure we are aware of the change.

[martinsuchan]

Hi, we just discovered when trying to upgrade to version 3.4.3 that this problem is actually not fixed.
We have a test method with following Data rows, and the build fails on non-CLS compliant warning/error.
Please add this variant to your CLS test suite, you should be able to reproduce the problem, thanks. @Evangelink

[DataRow("")]
[DataRow(null)]

[Evangelink]

I can only reproduce the issue with [DataRow(null)], I am investigating what I can do.

[martinsuchan]

Similarly this is now reported as non-CLS compliant:

[TestMethod]
[DataRow("some string", 1)]
public void StringIntDataRow(string s, int i)

Note it is possible that this non-CLS compliant behavior is the cause, or it's related to #3071 - if non-compliant tests are compiled to native code in UWP projects it might cause some troubles?