Test Coverage Reporting

[maennchen]

Changes

Implements test coverage reporting.

Requirements

• Can show coverage summary in CI
• Offers details for download (HTML?)
• Can show coverage summary in CLI
• Follows same rules as mix test.coverage for reporting (generated lines etc.)

TODOs / Issues

• mix test.coverage Summaries depend on mix being present. Find a way to avoid duplicating code.
• Some test break (Like FunctionClauseError message)
• Poluted IO from https://github.com/erlang/otp/blob/05737d130706c7189a8e6750d9c2252d2cc7987e/lib/tools/src/cover.erl#L1896

[maennchen]

@josevalim Do you have any preference on how to deal with the FunctionClauseError issues?

(See https://github.com/elixir-lang/elixir/actions/runs/13929976188/job/38984368756?pr=14343)

[josevalim]

@maennchen probably the best option is to check if coverage is enabled and perform only a partial assertion (i.e. break this assertion in two), the second one wrapped by a conditional.

[maennchen]

@josevalim Ok, I will do that. Do you btw. know the reason why this doesn't work while cover compiled?

[josevalim]

@josevalim Ok, I will do that. Do you btw. know the reason why this doesn't work while cover compiled?

We traverse the stored ast when analyzing the error message but when cover compiling the information is lost.

[maennchen]

@josevalim I got it down to one test failure:

  1) test structs in lists (ExUnit.DiffTest)
     test/ex_unit/diff_test.exs:943
     Assertion with =~ failed
     code:  assert diff_left =~ expected_left
     left:  "[\n  -%ExUnit.DiffTest.Customer{\n    address: %{\"street\" => \"123 Main St\", \"zip\" => \"62701\"},\n    language: \"en-US\",\n    age: 30,\n    first_name: \"John\",\n    last_name: \"Doe\",\n    notifications: true\n  }-\n]"
     right: "[\n  -%ExUnit.DiffTest.Customer{\n    address: %{\"street\" => \"123 Main St\", \"zip\" => \"62701\"},\n    age: 30,\n    first_name: \"John\",\n    language: \"en-US\",\n    last_name: \"Doe\",\n    notifications: true\n  }-\n]"
     stacktrace:
       test/ex_unit/diff_test.exs:123: ExUnit.DiffTest.refute_diff/5
       test/ex_unit/diff_test.exs:956: (test)

The ExUnit.DiffTest.Customer module is defined inside the test and therefore should also not be cover compiled. Do you have a clue why the order is different?

[josevalim]

@maennchen i would have to investigate locally which order is being printed. It seems it was supposed to be alphabetical.

[maennchen]

Splitting PR into multiple PRs. Marking as draft until done.

[josevalim]

Is this a left-over?

[maennchen]

No, this is intentionally as a dependency of the cover target.

The idea is that you can call make cover and it will run all the tests with coverage. (I tried to make it simple for non-regular contributors as well to get the coverage report.)

[josevalim]

Thank you, codewise, this looks good.

We just need to improve the quality of the report a bit. For this PR, I think the only thing we need to do is to exclude deprecated modules from the report. Is this possible? In theory we should be able to filter by this:

match? {:docs_v1, _, _, _, _, %{deprecated: _}}, Code.fetch_docs(mod)

But I don't know if the best moment is when running tests or when assembling coverage. I would also skip the Kernel module, because most of it is inlined by the compiler anyway.

The other tests we should improve the coverage through separate pull requests.

What are your thoughts?

[maennchen]

On Deprecation: I'm trying it out. I would directly exclude them from cover compiling. At that stage I should still have access to the unmodified AST including docs.

On the Kernel: Looking through the HTML, I think it still valuable to include it in the reporting. Stuff like this is helpful when writing tests and the coverage is at 80% which isn't that low.:

However with Kernel.SpecialForms, we can make an exception. I don't think that module is called anywhere.

I would however prefer to draw the line after the deprecation exclusion for this specific PR. The rest I would do in separate PRs as you propose.

[maennchen]

PR is updated with exclusion of deprecated modules.

[josevalim]

💚 💙 💜 💛 ❤️

[josevalim]

You are right. I did a mistake between Kernel.CLI and Kernel. Kernel.CLI is tested... but it uses separate OS processes, so it does not get measured. :)

[maennchen]

FYI on the OpenSSF Best Practices Badge:

There's some requirements in the higher levels (Silver / Gold). While I don't suggest to try to fulfill them at this time, this may be relevant to prepare for it a bit longer term.

Silver:

• The project MUST have FLOSS automated test suite(s) that provide at least 80% statement coverage if there is at least one FLOSS tool that can measure this criterion in the selected language. [test_statement_coverage80]

Gold:

• The project MUST have FLOSS automated test suite(s) that provide at least 90% statement coverage if there is at least one FLOSS tool that can measure this criterion in the selected language. [test_statement_coverage90]
• The project MUST have FLOSS automated test suite(s) that provide at least 80% branch coverage if there is at least one FLOSS tool that can measure this criterion in the selected language. [test_branch_coverage80]