pytest discovery 30x longer for VS Code than in a regular terminal

[ClementWalter]

Type: Performance Issue

The repository is open source at github.com/kkrt-labs/kakarot.

Just configure pytest as the test runner, with tests as the base directory. The logs in the output tab show

==== 1156 tests collected in 27.45s ====

Using only the native pytest command directly in a VS Code returns

==== 1156 tests collected in 0.87s ====

VS Code version: Code 1.88.1 (e170252f762678dec6ca2cc69aba1570769a5d39, 2024-04-10T17:43:08.196Z)
OS version: Darwin arm64 23.5.0
Modes:

System Info

Item	Value
CPUs	Apple M2 Max (12 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: enabled webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	4, 4, 4
Memory (System)	64.00GB (18.28GB free)
Process Argv	. --crash-reporter-id 8fb28320-a685-4761-a70f-8fd3d2132a10
Screen Reader	yes
VM	0%

Process Info

CPU %	Mem MB	   PID	Process
    7	   262	 12696	code main
    0	    66	 12771	   gpu-process
    0	    66	 12772	   utility-network-service
    0	   590	 12773	window [1] (conftest.py — kakarot)
    0	    66	 12774	ptyHost
    0	     0	 12789	     zsh (qterm)
    0	     0	 12919	       /bin/zsh --login
    0	     0	 12790	     zsh (qterm)
    0	     0	 12920	       /bin/zsh --login
    0	     0	 12791	     zsh (qterm)
    0	     0	 12918	       /bin/zsh --login
    0	     0	 12948	     zsh (qterm)
    0	     0	 13013	       /bin/zsh
    0	     0	 13286	     zsh (qterm)
    0	     0	 13358	       /bin/zsh
    0	     0	 13661	     zsh (qterm)
    0	     0	 13754	       /bin/zsh --login
    0	   721	 12775	extensionHost [1]
    0	    66	 12909	     /Users/clementwalter/.pyenv/versions/3.10.13/envs/kakarot/bin/python /Users/clementwalter/.vscode/extensions/ms-python.isort-2023.13.11511010/bundled/tool/lsp_server.py
    0	   262	 12947	     /Users/clementwalter/.pyenv/versions/3.10.13/envs/kakarot/bin/python /Users/clementwalter/.vscode/extensions/ms-python.python-2024.4.1/python_files/run-jedi-language-server.py
    0	    66	 13339	       /Users/clementwalter/.pyenv/versions/3.10.13/envs/kakarot/bin/python /Users/clementwalter/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/jedilsp/jedi/inference/compiled/subprocess/__main__.py /Users/clementwalter/.vscode/extensions/ms-python.python-2024.4.1/python_files/lib/jedilsp 3.10.13
    0	    66	 13571	     electron-nodejs (server.js )
    0	    66	 13575	     /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin) /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/json-language-features/server/dist/node/jsonServerMain --node-ipc --clientProcessId=12775
    0	    66	 15195	     electron-nodejs (server-node.js )
    0	    66	 15465	     /Users/clementwalter/.cache/trunk/cli/1.19.0-darwin-arm64/trunk lsp-proxy
    0	   262	 15592	     electron-nodejs (server.js )
    0	   131	 12776	shared-process
    0	     0	 27183	     /bin/ps -ax -o pid=,ppid=,pcpu=,pmem=,command=
    0	    66	 12777	fileWatcher [1]
    0	   131	 27144	window [2] (Issue Reporter)

Workspace Info

|  Window (conftest.py — kakarot)
|    Folder (kakarot): more than 21995 files
|      File types: json(19458) sol(126) cairo(61) md(16) py(16) yml(11)
|                  pyc(11) gitignore(10) toml(10) xml(6)
|      Conf files: github-actions(4) package.json(3) makefile(2)
|                  devcontainer.json(1) launch.json(1) settings.json(1)
|                  dockerfile(1)
|      Launch Configs: debugpy(2);

Extensions (46)

Extension	Author (truncated)	Version
rust-bundle	1Yi	1.0.0
codewhisperer-for-command-line-companion	ama	1.3.2
markdown-mermaid	bie	1.23.1
markdown-preview-github-styles	bie	2.0.4
scratchpads	bue	1.0.0
rust-syntax	dus	0.6.1
gitlens	eam	15.2.0
cairo-ls	eri	0.0.28
prettier-vscode	esb	10.4.0
copilot	Git	1.214.997
copilot-chat	Git	0.14.1
vscode-github-actions	git	0.26.3
vscode-pull-request-github	Git	0.86.1
gc-excelviewer	Gra	4.2.59
todo-tree	Gru	0.0.226
vscode-test-explorer	hbe	2.21.1
bisectdemo	jri	0.0.3
solidity	Jua	0.0.174
vscode-gist	ken	3.0.3
rainbow-csv	mec	3.12.0
dotenv	mik	1.0.1
vscode-docker	ms-	1.29.1
debugpy	ms-	2024.9.12011012
isort	ms-	2023.13.11511010
python	ms-	2024.4.1
jupyter	ms-	2024.4.2024041101
jupyter-keymap	ms-	1.1.2
jupyter-renderers	ms-	1.0.18
vscode-jupyter-cell-tags	ms-	0.1.9
vscode-jupyter-slideshow	ms-	0.1.6
remote-containers	ms-	0.375.1
cmake-tools	ms-	1.18.43
hexeditor	ms-	1.9.14
makefile-tools	ms-	0.9.10
test-adapter-converter	ms-	0.1.9
vscode-typescript-next	ms-	5.6.20240721
vsliveshare	ms-	1.0.5936
vscode-xml	red	0.27.1
rust-analyzer	rus	0.4.2044
cairo	Sta	0.10.0
even-better-toml	tam	0.19.2
vscode-vyper	tin	0.1.0
io	tru	3.18.0
cmake	twx	0.0.17
volar	Vue	2.0.8
debug	web	0.27.0

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonnoceb:30805159
asynctok:30898717
pythonregdiag2:30936856
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
accentitlementst:30995554
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
bdiig495:31013172
a69g1124:31058053
dvdeprecation:31068756
dwnewjupyter:31046869
newcmakeconfigv2:31071590
impr_priority:31094925
refactort:31098051
4930e709:31092288

[vscodenpa]

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.91.1. Please try upgrading to the latest version and checking whether this issue remains.

Happy Coding!

[eleanorjboyd]

Hi! Sorry for the delay here. I am honestly not sure what could be impacting this since all we do is spin up a subprocess that runs pytest.main with the same args. This is the link to the code with the args that are then run https://github.com/eleanorjboyd/vscode-python/blob/776301661e845ebd38cf007a2eddf800b830749c/src/client/testing/testController/pytest/pytestDiscoveryAdapter.ts#L110-L111. I think it could just be running it in a node subprocess could mean it has less memory or no parallelism? @karthiknadig any thoughts here?

[karthiknadig]

Looks like they have a repro here: github.com/kkrt-labs/kakarot that you can try.

[github-actions]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.

Happy Coding!

[ClementWalter]

This has been solved with a previous (can't remember which) version of VS Code, thanks