WIP: Setup unit test code coverage using nyc

[andyleejordan]

Resolves #3597.

[andyleejordan]

Ugh this is being annoying.

[andyleejordan]

So @rjmholt nyc does not currently work on Windows. There's a myriad of bug reports about it, and they haven't pushed a new release in over a year. It is very sad.

istanbuljs/nyc#1381 repros on my Surface, without any symlinks. Looks like they messed up path.join vs path.resolve and the casing that's returned for root drive letters causes the case sensitive globs to fail.

I'll split out a couple things from this and leave it WIP I guess. Sucks, I can't find another tool for JavaScript code coverage. @TylerLeonhardt do you have any suggestions?

[andyleejordan]

Ok so at this point we are punting again. This branch produces a coverage report on my local macMini:

----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s                                                                                                                    
----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
All files                   |   45.71 |    24.09 |   34.78 |   49.93 |                                                                                                                                      
 src                        |   56.54 |    32.31 |   54.86 |   61.05 |                                                                                                                                      
  debugAdapter.ts           |    82.6 |    64.28 |   63.63 |   81.81 | 36,46,54-55,60-61,68,96                                                                                                              
  languageClientConsumer.ts |   58.33 |       25 |      50 |   71.42 | 19-22                                                                                                                                
  logging.ts                |   45.28 |    30.76 |   44.82 |      48 | 49-58,93-133,154-155,157-168                                                                                                         
  main.ts                   |    86.2 |    46.15 |      60 |    86.2 | 197-217                                                                                                                              
  platform.ts               |   35.15 |    18.37 |   55.81 |      40 | 45,48-49,118-131,151-152,174-179,198-211,221,234,247,263-288,303-464,469                                                             
  process.ts                |   87.93 |    53.57 |   71.42 |   87.27 | 52,131,149,175,204-210                                                                                                               
  session.ts                |   53.29 |     39.2 |   45.83 |   53.29 | ...8,196,216-245,265,290-320,337-338,345-365,393-409,420-421,450,485-499,535-539,557-559,590,606-624,682-686,692-700,717-786,794-796 
  settings.ts               |   95.06 |    82.14 |   86.66 |   94.23 | 287,293,301                                                                                                                          
  utils.ts                  |   61.62 |       25 |   57.69 |   77.35 | 21,33-38,44-49,86,106,113-114                                                                                                        
 src/controls               |   24.52 |        0 |       0 |   24.13 |                                                                                                                                      
  checkboxQuickPick.ts      |   24.52 |        0 |       0 |   24.13 | 27-96                                                                                                                                
 src/features               |   38.26 |    17.59 |   24.53 |    42.4 |                                                                                                                                      
  CodeActions.ts            |   35.71 |        0 |   16.66 |   35.71 | 14-44                                                                                                                                
  Console.ts                |   29.19 |    16.66 |   17.77 |   38.82 | 89-193,207-250,255-264,291-292,299                                                                                                   
  CustomViews.ts            |   31.54 |    27.27 |   15.78 |   45.65 | 18-25,33-64,81-137,169-173,204-233                                                                                                   
  DebugSession.ts           |   32.81 |    13.76 |   22.72 |   32.28 | 61-111,165-168,185-212,251-298,348-372,410-411,425-510,540-541,555-630                                                               
  Examples.ts               |   71.42 |      100 |   33.33 |   71.42 | 15-23                                                                                                                                
  ExpandAlias.ts            |   47.05 |        0 |      20 |   47.05 | 19-45                                                                                                                                
  ExtensionCommands.ts      |   20.43 |     5.63 |    4.54 |   21.21 | 54-86,191-227,239-625                                                                                                                
  ExternalApi.ts            |    93.1 |     62.5 |   85.71 |      92 | 66,71                                                                                                                                
  FindModule.ts             |      28 |        0 |      10 |      28 | 24-93                                                                                                                                
  GenerateBugReport.ts      |   42.85 |        0 |      25 |   40.74 | 18,34-118                                                                                                                            
  GetCommands.ts            |    65.9 |    77.77 |   57.14 |   65.11 | 35,48-52,64-65,69,77-82,132-139                                                                                                      
  HelpCompletion.ts         |   41.17 |    15.78 |   44.44 |      40 | 45,63-69,95-128,146,154-193                                                                                                          
  ISECompatibility.ts       |   54.83 |     37.5 |      50 |   63.15 | 34-40,51-53,60-61                                                                                                                    
  NewFileOrProject.ts       |   33.33 |        4 |   11.76 |   33.33 | 20-52,65-167                                                                                                                         
  OpenInISE.ts              |      50 |        0 |   33.33 |      50 | 13-32                                                                                                                                
  PesterTests.ts            |   75.86 |    54.54 |   61.53 |   75.86 | 33-44,61-65                                                                                                                          
  RemoteFiles.ts            |   68.42 |       40 |   55.55 |   72.22 | 52,65-70                                                                                                                             
  RunCode.ts                |   51.21 |    35.29 |      25 |      60 | 23-52,63                                                                                                                             
  ShowHelp.ts               |   43.75 |        0 |   33.33 |   43.75 | 18-35                                                                                                                                
  UpdatePowerShell.ts       |   34.09 |     8.97 |   27.77 |   42.85 | 33-36,47-48,53,67-71,105-190                                                                                                         
----------------------------|---------|----------|---------|---------|--------------------------------------------------------------------------------------------------------------------------------------
nyc
│   56.98 % Lines
└─┬ /usr/local/bin/node /Users/andschwa/src/vscode-powershell/out/test/runTests.js
  │   56.98 % Lines
  └─┬ /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/MacOS/Electron --disable-extensions ./test --no-sandbox --disable-workspace-trust --extensionDevelopmentPath=/Users/andschwa/src/vscode-powershell --extensionTestsPath=/Users/andschwa/src/vscode-powershell/out/test/index --crash-reporter-id 711b0fb3-33c9-4e09-a2f3-289124f3fa1e
    │   56.34 % Lines
    └─┬ /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/MacOS/Electron
      │   56.34 % Lines
      └── /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Renderer).app/Contents/MacOS/Code - Insiders Helper (Renderer) /Users/andschwa/src/vscode-powershell/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-fork --type=extensionHost --skipWorkspaceStorageLock
            56.34 % Lines

And so it should work in CI, but it doesn't. It only runs if we set use-spawn-wrap to true (which is being deprecated anyway) and then it doesn't actually produce any results because it fails to analyze files touched by the subprocesses the Code extension test suite executes.

I am far from the only person struggling with nyc (which is unfortunate given it's the code coverage tool for JavaScript). Here's a prior Code C# / OmniSharp extension maintainer having issues: istanbuljs/nyc#728

And despite eventually getting it working, the implementation in that extension's CI was eventually removed: dotnet/vscode-csharp#4506 (comment)

To quote the maintainer directly:

I remember there being a lot of difficulties getting it to work as expected when moving from the vscode to the vscode-test package. The effort to get it working wasn't justified by the value we got from it.

While an overview of the Code Python extension looks like it uses nyc (and indeed I followed their setup initially) it turns out that they have their own test runner, and thus are able to execute their JavaScript unit tests directly on Ubuntu without instantiating Code. Indeed, they do not even install the @vscode/test-electron package which most extension's use to launch Code (and then it launches the tests within Code, hence the subprocesses). So it's not actually a useful example for us.

Now theoretically, if we want to take one more try at this, we could pre-instrument the codebase dynamically after the test Code instance is launched by having Mocha use Babel to transpile the JavaScript and babel-plugin-istanbul to instrument the code, and then maybe nyc would work. But there's no guarantees and it seems, in a word, difficult. This was suggested here.

Also here two Code extension with coverage examples, but they're out of date as far as I can tell:

• https://github.com/codecov/example-typescript-vscode-extension
• https://github.com/jedwards1211/vscode-extension-skeleton

[andyleejordan]

I don't think I'll get back to using Istanbul, it's sooo out of date. We'll do this another way.