Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test coverage as a merge gate #10800

Open
stevenctl opened this issue Mar 10, 2025 · 0 comments
Open

test coverage as a merge gate #10800

stevenctl opened this issue Mar 10, 2025 · 0 comments

Comments

@stevenctl
Copy link
Contributor

stevenctl commented Mar 10, 2025

I'd like to open up a discussion on the fact that we use this as a gate to merge code. It's a good metric to see which areas have tests or not, but it doesn't give much signal about test quality. Making it a target is what I take issue with... this does not encourage 1) writing quality tests 2) ensuring code is correct. In fact, I think it encourages writing tests that focus more on structure than behavior.

Regardless of whether we remove it as a gate.. can we at least improve the output so it's easy to understand what the targets are? When I get a failed job on my PR, I see some lines like

2025-03-10T22:47:14.5979754Z 	github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway/testutils		coverage: 0.0% of statements

Most are package I didn't touch, or things like wellknown which are pretty much just a collection of constants or insanely simple oneliner functions. I have no idea what the targets are without inspecting our build configs, or what in my PR is missing. There are packages with 0.0 or very little that my PR doesn't touch but also aren't excluded in the test_coverage.yml.

``` 2025-03-10T22:42:19.9577642Z ##[group]Run make go-test-with-coverage 2025-03-10T22:42:19.9577981Z �[36;1mmake go-test-with-coverage�[0m 2025-03-10T22:42:19.9633941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-10T22:42:19.9634252Z env: 2025-03-10T22:42:19.9634436Z VERSION: 1.0.0-ci1 2025-03-10T22:42:19.9635086Z GITHUB_TOKEN: *** 2025-03-10T22:42:19.9635326Z TEST_PKG: ./internal/kgateway/... 2025-03-10T22:42:19.9635570Z ##[endgroup] 2025-03-10T22:42:20.1033344Z rm -rf /home/runner/work/kgateway/kgateway/_test/bug_report 2025-03-10T22:42:20.1042250Z rm -rf /home/runner/work/kgateway/kgateway/_test/test_log 2025-03-10T22:42:20.1051176Z mkdir -p /home/runner/work/kgateway/kgateway/_test/bug_report 2025-03-10T22:42:20.1063469Z mkdir -p /home/runner/work/kgateway/kgateway/_test/test_log 2025-03-10T22:47:14.5926219Z go: downloading github.com/stretchr/testify v1.10.0 2025-03-10T22:47:14.5926891Z go: downloading github.com/fgrosse/zaptest v1.1.0 2025-03-10T22:47:14.5927792Z go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 2025-03-10T22:47:14.5928890Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/admin 1.234s coverage: 9.4% of statements 2025-03-10T22:47:14.5930153Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/controller 12.740s coverage: 51.0% of statements 2025-03-10T22:47:14.5931358Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/crds coverage: 0.0% of statements 2025-03-10T22:47:14.5932300Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/deployer 2.024s coverage: 61.4% of statements 2025-03-10T22:47:14.5933092Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/endpoints coverage: 0.0% of statements 2025-03-10T22:47:14.5934125Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2 [no test files] 2025-03-10T22:47:14.5935480Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/common coverage: 0.0% of statements 2025-03-10T22:47:14.5936721Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugin coverage: 0.0% of statements 2025-03-10T22:47:14.5938004Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/backend coverage: 0.0% of statements 2025-03-10T22:47:14.5939427Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/backend/ai 1.189s coverage: 62.4% of statements 2025-03-10T22:47:14.5940879Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/destrule coverage: 0.0% of statements 2025-03-10T22:47:14.5942278Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/directresponse coverage: 0.0% of statements 2025-03-10T22:47:14.5943191Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/httplistenerpolicy 1.203s coverage: 44.6% of statements 2025-03-10T22:47:14.5944065Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/istio coverage: 0.0% of statements 2025-03-10T22:47:14.5945056Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/kubernetes coverage: 0.0% of statements 2025-03-10T22:47:14.5945922Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/listenerpolicy coverage: 0.0% of statements 2025-03-10T22:47:14.5946790Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/routepolicy 1.184s coverage: 30.0% of statements 2025-03-10T22:47:14.5947954Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/sandwich coverage: 0.0% of statements 2025-03-10T22:47:14.5948773Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/waypoint 6.537s coverage: 80.0% of statements 2025-03-10T22:47:14.5949685Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/waypoint/waypointquery coverage: 0.0% of statements 2025-03-10T22:47:14.5950535Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/pluginutils coverage: 0.0% of statements 2025-03-10T22:47:14.5951468Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/registry coverage: 0.0% of statements 2025-03-10T22:47:14.5952227Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/settings 1.041s coverage: 100.0% of statements 2025-03-10T22:47:14.5952805Z make: *** [Makefile:236: go-test] Error 1 2025-03-10T22:47:14.5953289Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/filters coverage: 0.0% of statements 2025-03-10T22:47:14.5953850Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/helm [no test files] 2025-03-10T22:47:14.5954428Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/ir coverage: 0.0% of statements 2025-03-10T22:47:14.5955251Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/krtcollections 2.531s coverage: 50.1% of statements 2025-03-10T22:47:14.5955891Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/parameters [no test files] 2025-03-10T22:47:14.5956533Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/plugins coverage: 0.0% of statements 2025-03-10T22:47:14.5957176Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/ports 1.028s coverage: 100.0% of statements 2025-03-10T22:47:14.5957856Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/proxy_syncer 1.206s coverage: 17.4% of statements 2025-03-10T22:47:14.5958496Z Running Suite: Query Suite - /home/runner/work/kgateway/kgateway/internal/kgateway/query 2025-03-10T22:47:14.5958939Z ======================================================================================== 2025-03-10T22:47:14.5959480Z Random Seed: �[1m1741646744�[0m 2025-03-10T22:47:14.5959634Z 2025-03-10T22:47:14.5959781Z Will run �[1m1�[0m of �[1m29�[0m specs 2025-03-10T22:47:14.5960248Z �[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m2025-03-10T22:45:44.524423Z info krt policies synced 2025-03-10T22:47:14.5960956Z 2025-03-10T22:45:44.524638Z info krt ManyCollection[*v1.Service,ir.BackendObjectIR] synced owner=ManyCollection[*v1.Service,ir.BackendObjectIR] 2025-03-10T22:47:14.5961888Z 2025-03-10T22:45:44.524837Z info krt Collection[ir.BackendObjectIR,ir.BackendObjectIR] synced owner=Collection[ir.BackendObjectIR,ir.BackendObjectIR] 2025-03-10T22:47:14.5962706Z 2025-03-10T22:45:44.524614Z info krt Collection[*v1.Secret,ir.Secret] synced owner=Collection[*v1.Secret,ir.Secret] 2025-03-10T22:47:14.5963305Z 2025-03-10T22:45:44.525026Z info krt NamespacesMetadata synced owner=NamespacesMetadata 2025-03-10T22:47:14.5963921Z 2025-03-10T22:45:44.525157Z info krt routes-tls-routes-with-policy synced owner=routes-tls-routes-with-policy 2025-03-10T22:47:14.5964962Z 2025-03-10T22:45:44.525193Z info krt http-routes-with-policy synced owner=http-routes-with-policy 2025-03-10T22:47:14.5966064Z 2025-03-10T22:45:44.525223Z info krt routes-tcp-routes-with-policy synced owner=routes-tcp-routes-with-policy 2025-03-10T22:47:14.5967254Z 2025-03-10T22:45:44.527566Z info krt routes-http-routes-with-policy synced owner=routes-http-routes-with-policy 2025-03-10T22:47:14.5968199Z 2025-03-10T22:45:44.527649Z info krt all-routes-with-policy synced 2025-03-10T22:47:14.5971907Z �[38;5;10m•�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m 2025-03-10T22:47:14.5972808Z 2025-03-10T22:47:14.5973052Z �[38;5;10m�[1mRan 1 of 29 Specs in 0.105 seconds�[0m 2025-03-10T22:47:14.5973690Z �[38;5;10m�[1mSUCCESS!�[0m -- �[38;5;10m�[1m1 Passed�[0m | �[38;5;9m�[1m0 Failed�[0m | �[38;5;11m�[1m0 Pending�[0m | �[38;5;14m�[1m28 Skipped�[0m 2025-03-10T22:47:14.5974141Z PASS | FOCUSED 2025-03-10T22:47:14.5974473Z FAIL github.com/kgateway-dev/kgateway/v2/internal/kgateway/query 0.319s 2025-03-10T22:47:14.5975301Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/query/mocks coverage: 0.0% of statements 2025-03-10T22:47:14.5976120Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/reports 1.116s coverage: 90.0% of statements 2025-03-10T22:47:14.5976800Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/setup 81.443s coverage: 50.6% of statements 2025-03-10T22:47:14.5977467Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator coverage: 0.0% of statements 2025-03-10T22:47:14.5978191Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/backendref 1.024s coverage: 11.1% of statements 2025-03-10T22:47:14.5978962Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway 39.885s coverage: 84.2% of statements 2025-03-10T22:47:14.5979754Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway/testutils coverage: 0.0% of statements 2025-03-10T22:47:14.5980528Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/httproute 1.195s coverage: 37.2% of statements 2025-03-10T22:47:14.5981309Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/irtranslator 1.292s coverage: 4.6% of statements 2025-03-10T22:47:14.5982077Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/listener 1.153s coverage: 50.6% of statements 2025-03-10T22:47:14.5982842Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/routeutils 1.125s coverage: 66.7% of statements 2025-03-10T22:47:14.5983711Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/sslutils coverage: 0.0% of statements 2025-03-10T22:47:14.5984453Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/utils coverage: 0.0% of statements 2025-03-10T22:47:14.5985363Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/utils 1.021s coverage: 8.3% of statements 2025-03-10T22:47:14.5986031Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/utils/krtutil coverage: 0.0% of statements 2025-03-10T22:47:14.5986684Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/wellknown coverage: 0.0% of statements 2025-03-10T22:47:14.5987306Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/xds coverage: 0.0% of statements 2025-03-10T22:47:14.5987695Z FAIL 2025-03-10T22:47:14.5991680Z ##[error]Process completed with exit code 2. ```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant