You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'd like to open up a discussion on the fact that we use this as a gate to merge code. It's a good metric to see which areas have tests or not, but it doesn't give much signal about test quality. Making it a target is what I take issue with... this does not encourage 1) writing quality tests 2) ensuring code is correct. In fact, I think it encourages writing tests that focus more on structure than behavior.
Regardless of whether we remove it as a gate.. can we at least improve the output so it's easy to understand what the targets are? When I get a failed job on my PR, I see some lines like
2025-03-10T22:47:14.5979754Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway/testutils coverage: 0.0% of statements
Most are package I didn't touch, or things like wellknown which are pretty much just a collection of constants or insanely simple oneliner functions. I have no idea what the targets are without inspecting our build configs, or what in my PR is missing. There are packages with 0.0 or very little that my PR doesn't touch but also aren't excluded in the test_coverage.yml.
```
2025-03-10T22:42:19.9577642Z ##[group]Run make go-test-with-coverage
2025-03-10T22:42:19.9577981Z �[36;1mmake go-test-with-coverage�[0m
2025-03-10T22:42:19.9633941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-03-10T22:42:19.9634252Z env:
2025-03-10T22:42:19.9634436Z VERSION: 1.0.0-ci1
2025-03-10T22:42:19.9635086Z GITHUB_TOKEN: ***
2025-03-10T22:42:19.9635326Z TEST_PKG: ./internal/kgateway/...
2025-03-10T22:42:19.9635570Z ##[endgroup]
2025-03-10T22:42:20.1033344Z rm -rf /home/runner/work/kgateway/kgateway/_test/bug_report
2025-03-10T22:42:20.1042250Z rm -rf /home/runner/work/kgateway/kgateway/_test/test_log
2025-03-10T22:42:20.1051176Z mkdir -p /home/runner/work/kgateway/kgateway/_test/bug_report
2025-03-10T22:42:20.1063469Z mkdir -p /home/runner/work/kgateway/kgateway/_test/test_log
2025-03-10T22:47:14.5926219Z go: downloading github.com/stretchr/testify v1.10.0
2025-03-10T22:47:14.5926891Z go: downloading github.com/fgrosse/zaptest v1.1.0
2025-03-10T22:47:14.5927792Z go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
2025-03-10T22:47:14.5928890Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/admin 1.234s coverage: 9.4% of statements
2025-03-10T22:47:14.5930153Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/controller 12.740s coverage: 51.0% of statements
2025-03-10T22:47:14.5931358Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/crds coverage: 0.0% of statements
2025-03-10T22:47:14.5932300Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/deployer 2.024s coverage: 61.4% of statements
2025-03-10T22:47:14.5933092Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/endpoints coverage: 0.0% of statements
2025-03-10T22:47:14.5934125Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2 [no test files]
2025-03-10T22:47:14.5935480Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/common coverage: 0.0% of statements
2025-03-10T22:47:14.5936721Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugin coverage: 0.0% of statements
2025-03-10T22:47:14.5938004Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/backend coverage: 0.0% of statements
2025-03-10T22:47:14.5939427Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/backend/ai 1.189s coverage: 62.4% of statements
2025-03-10T22:47:14.5940879Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/destrule coverage: 0.0% of statements
2025-03-10T22:47:14.5942278Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/directresponse coverage: 0.0% of statements
2025-03-10T22:47:14.5943191Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/httplistenerpolicy 1.203s coverage: 44.6% of statements
2025-03-10T22:47:14.5944065Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/istio coverage: 0.0% of statements
2025-03-10T22:47:14.5945056Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/kubernetes coverage: 0.0% of statements
2025-03-10T22:47:14.5945922Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/listenerpolicy coverage: 0.0% of statements
2025-03-10T22:47:14.5946790Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/routepolicy 1.184s coverage: 30.0% of statements
2025-03-10T22:47:14.5947954Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/sandwich coverage: 0.0% of statements
2025-03-10T22:47:14.5948773Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/waypoint 6.537s coverage: 80.0% of statements
2025-03-10T22:47:14.5949685Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/plugins/waypoint/waypointquery coverage: 0.0% of statements
2025-03-10T22:47:14.5950535Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/pluginutils coverage: 0.0% of statements
2025-03-10T22:47:14.5951468Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/registry coverage: 0.0% of statements
2025-03-10T22:47:14.5952227Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/extensions2/settings 1.041s coverage: 100.0% of statements
2025-03-10T22:47:14.5952805Z make: *** [Makefile:236: go-test] Error 1
2025-03-10T22:47:14.5953289Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/filters coverage: 0.0% of statements
2025-03-10T22:47:14.5953850Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/helm [no test files]
2025-03-10T22:47:14.5954428Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/ir coverage: 0.0% of statements
2025-03-10T22:47:14.5955251Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/krtcollections 2.531s coverage: 50.1% of statements
2025-03-10T22:47:14.5955891Z ? github.com/kgateway-dev/kgateway/v2/internal/kgateway/parameters [no test files]
2025-03-10T22:47:14.5956533Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/plugins coverage: 0.0% of statements
2025-03-10T22:47:14.5957176Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/ports 1.028s coverage: 100.0% of statements
2025-03-10T22:47:14.5957856Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/proxy_syncer 1.206s coverage: 17.4% of statements
2025-03-10T22:47:14.5958496Z Running Suite: Query Suite - /home/runner/work/kgateway/kgateway/internal/kgateway/query
2025-03-10T22:47:14.5958939Z ========================================================================================
2025-03-10T22:47:14.5959480Z Random Seed: �[1m1741646744�[0m
2025-03-10T22:47:14.5959634Z
2025-03-10T22:47:14.5959781Z Will run �[1m1�[0m of �[1m29�[0m specs
2025-03-10T22:47:14.5960248Z �[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m2025-03-10T22:45:44.524423Z info krt policies synced
2025-03-10T22:47:14.5960956Z 2025-03-10T22:45:44.524638Z info krt ManyCollection[*v1.Service,ir.BackendObjectIR] synced owner=ManyCollection[*v1.Service,ir.BackendObjectIR]
2025-03-10T22:47:14.5961888Z 2025-03-10T22:45:44.524837Z info krt Collection[ir.BackendObjectIR,ir.BackendObjectIR] synced owner=Collection[ir.BackendObjectIR,ir.BackendObjectIR]
2025-03-10T22:47:14.5962706Z 2025-03-10T22:45:44.524614Z info krt Collection[*v1.Secret,ir.Secret] synced owner=Collection[*v1.Secret,ir.Secret]
2025-03-10T22:47:14.5963305Z 2025-03-10T22:45:44.525026Z info krt NamespacesMetadata synced owner=NamespacesMetadata
2025-03-10T22:47:14.5963921Z 2025-03-10T22:45:44.525157Z info krt routes-tls-routes-with-policy synced owner=routes-tls-routes-with-policy
2025-03-10T22:47:14.5964962Z 2025-03-10T22:45:44.525193Z info krt http-routes-with-policy synced owner=http-routes-with-policy
2025-03-10T22:47:14.5966064Z 2025-03-10T22:45:44.525223Z info krt routes-tcp-routes-with-policy synced owner=routes-tcp-routes-with-policy
2025-03-10T22:47:14.5967254Z 2025-03-10T22:45:44.527566Z info krt routes-http-routes-with-policy synced owner=routes-http-routes-with-policy
2025-03-10T22:47:14.5968199Z 2025-03-10T22:45:44.527649Z info krt all-routes-with-policy synced
2025-03-10T22:47:14.5971907Z �[38;5;10m•�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m�[38;5;14mS�[0m
2025-03-10T22:47:14.5972808Z
2025-03-10T22:47:14.5973052Z �[38;5;10m�[1mRan 1 of 29 Specs in 0.105 seconds�[0m
2025-03-10T22:47:14.5973690Z �[38;5;10m�[1mSUCCESS!�[0m -- �[38;5;10m�[1m1 Passed�[0m | �[38;5;9m�[1m0 Failed�[0m | �[38;5;11m�[1m0 Pending�[0m | �[38;5;14m�[1m28 Skipped�[0m
2025-03-10T22:47:14.5974141Z PASS | FOCUSED
2025-03-10T22:47:14.5974473Z FAIL github.com/kgateway-dev/kgateway/v2/internal/kgateway/query 0.319s
2025-03-10T22:47:14.5975301Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/query/mocks coverage: 0.0% of statements
2025-03-10T22:47:14.5976120Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/reports 1.116s coverage: 90.0% of statements
2025-03-10T22:47:14.5976800Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/setup 81.443s coverage: 50.6% of statements
2025-03-10T22:47:14.5977467Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator coverage: 0.0% of statements
2025-03-10T22:47:14.5978191Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/backendref 1.024s coverage: 11.1% of statements
2025-03-10T22:47:14.5978962Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway 39.885s coverage: 84.2% of statements
2025-03-10T22:47:14.5979754Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/gateway/testutils coverage: 0.0% of statements
2025-03-10T22:47:14.5980528Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/httproute 1.195s coverage: 37.2% of statements
2025-03-10T22:47:14.5981309Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/irtranslator 1.292s coverage: 4.6% of statements
2025-03-10T22:47:14.5982077Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/listener 1.153s coverage: 50.6% of statements
2025-03-10T22:47:14.5982842Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/routeutils 1.125s coverage: 66.7% of statements
2025-03-10T22:47:14.5983711Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/sslutils coverage: 0.0% of statements
2025-03-10T22:47:14.5984453Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/translator/utils coverage: 0.0% of statements
2025-03-10T22:47:14.5985363Z ok github.com/kgateway-dev/kgateway/v2/internal/kgateway/utils 1.021s coverage: 8.3% of statements
2025-03-10T22:47:14.5986031Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/utils/krtutil coverage: 0.0% of statements
2025-03-10T22:47:14.5986684Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/wellknown coverage: 0.0% of statements
2025-03-10T22:47:14.5987306Z github.com/kgateway-dev/kgateway/v2/internal/kgateway/xds coverage: 0.0% of statements
2025-03-10T22:47:14.5987695Z FAIL
2025-03-10T22:47:14.5991680Z ##[error]Process completed with exit code 2.
```
The text was updated successfully, but these errors were encountered:
I'd like to open up a discussion on the fact that we use this as a gate to merge code. It's a good metric to see which areas have tests or not, but it doesn't give much signal about test quality. Making it a target is what I take issue with... this does not encourage 1) writing quality tests 2) ensuring code is correct. In fact, I think it encourages writing tests that focus more on structure than behavior.
Regardless of whether we remove it as a gate.. can we at least improve the output so it's easy to understand what the targets are? When I get a failed job on my PR, I see some lines like
Most are package I didn't touch, or things like
wellknown
which are pretty much just a collection of constants or insanely simple oneliner functions. I have no idea what the targets are without inspecting our build configs, or what in my PR is missing. There are packages with 0.0 or very little that my PR doesn't touch but also aren't excluded in the test_coverage.yml.The text was updated successfully, but these errors were encountered: