Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[POC] Flaky test retries #8113

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from
Draft

[POC] Flaky test retries #8113

wants to merge 8 commits into from
+62 −85

Conversation

smola
Copy link
Member

@smola smola commented Dec 18, 2024 •
edited
Loading

What Does This Do

  • Use Develocity Test Retry (docs) so retry flaky tests in CI.
  • Only tests marked as @Flaky will be retried.
  • Failed runs are visible in the logs, and preserved. This keeps observability of intermediate runs in Datadog Test Optimization.

Why?

This would allow to get closer to CI pipelines always succeeding on the first run. Our past strategy seggregating flaky test to a special CI job that never fails the pipeline has led to many tests regressing from flaky to broken.

While retrying can make some individual jobs slower, if we account for the overall time between pushing and manually retrying, the overall time spent in CI should be way lower.

Compared to the old @Retry approach, this plugin has the advantage of not hiding intermediate failed runs. So we should still be able to monitor flaky tests. in fact, given that flaky tests are always retried at least once, flaky test detection should be more accurate, both at Circle CI and Datadog Test Optimization.

Caveats

There are some limitations of this plugin:

  • @Flaky annotation only works at class level, not method level (feature request).
  • It does not support advanced runtime conditions, such as filtering by JVM vendor or version, or limiting retyring to specific errors (feature request).

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Dec 18, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/retry-flaky-tests
git_commit_date 1743173661 1743173178
git_commit_sha 86e5bec fdf8c7f
release_version 1.48.0-SNAPSHOT~86e5bec219 1.48.0-SNAPSHOT~fdf8c7f65e
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1743176445 1743176445
ci_job_id 870218826 870218826
ci_pipeline_id 60342509 60342509
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-3yvggzrb-project-304-concurrent-2-txj03hr1 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-3yvggzrb-project-304-concurrent-2-txj03hr1 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1060902
Total [baseline] (8.689 s) : 0, 8689194
Agent [candidate] (1.052 s) : 0, 1051742
Total [candidate] (8.666 s) : 0, 8666411
section iast
Agent [baseline] (1.178 s) : 0, 1177707
Total [baseline] (9.293 s) : 0, 9292696
Agent [candidate] (1.178 s) : 0, 1178072
Total [candidate] (9.289 s) : 0, 9289014
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.189 s) : 0, 1189086
Total [baseline] (9.222 s) : 0, 9222107
Agent [candidate] (1.177 s) : 0, 1177120
Total [candidate] (9.199 s) : 0, 9199259
section iast_TELEMETRY_OFF
Agent [baseline] (1.182 s) : 0, 1182197
Total [baseline] (9.235 s) : 0, 9235441
Agent [candidate] (1.173 s) : 0, 1172763
Total [candidate] (9.278 s) : 0, 9277710
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent iast 1.178 s 116.805 ms (11.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.189 s 128.184 ms (12.1%)
Agent iast_TELEMETRY_OFF 1.182 s 121.294 ms (11.4%)
Total tracing 8.689 s -
Total iast 9.293 s 603.502 ms (6.9%)
Total iast_HARDCODED_SECRET_DISABLED 9.222 s 532.913 ms (6.1%)
Total iast_TELEMETRY_OFF 9.235 s 546.247 ms (6.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.052 s -
Agent iast 1.178 s 126.33 ms (12.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.177 s 125.378 ms (11.9%)
Agent iast_TELEMETRY_OFF 1.173 s 121.02 ms (11.5%)
Total tracing 8.666 s -
Total iast 9.289 s 622.603 ms (7.2%)
Total iast_HARDCODED_SECRET_DISABLED 9.199 s 532.848 ms (6.1%)
Total iast_TELEMETRY_OFF 9.278 s 611.299 ms (7.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (724.836 ms) : 0, 724836
BytebuddyAgent [candidate] (719.289 ms) : 0, 719289
GlobalTracer [baseline] (241.143 ms) : 0, 241143
GlobalTracer [candidate] (239.151 ms) : 0, 239151
AppSec [baseline] (55.436 ms) : 0, 55436
AppSec [candidate] (54.591 ms) : 0, 54591
Debugger [baseline] (5.886 ms) : 0, 5886
Debugger [candidate] (5.182 ms) : 0, 5182
Remote Config [baseline] (708.165 µs) : 0, 708
Remote Config [candidate] (703.117 µs) : 0, 703
Telemetry [baseline] (12.957 ms) : 0, 12957
Telemetry [candidate] (13.088 ms) : 0, 13088
section iast
BytebuddyAgent [baseline] (837.326 ms) : 0, 837326
BytebuddyAgent [candidate] (836.778 ms) : 0, 836778
GlobalTracer [baseline] (228.881 ms) : 0, 228881
GlobalTracer [candidate] (228.89 ms) : 0, 228890
AppSec [baseline] (55.674 ms) : 0, 55674
AppSec [candidate] (56.191 ms) : 0, 56191
Debugger [baseline] (4.145 ms) : 0, 4145
Debugger [candidate] (4.242 ms) : 0, 4242
Remote Config [baseline] (606.877 µs) : 0, 607
Remote Config [candidate] (624.687 µs) : 0, 625
Telemetry [baseline] (8.803 ms) : 0, 8803
Telemetry [candidate] (8.907 ms) : 0, 8907
IAST [baseline] (22.644 ms) : 0, 22644
IAST [candidate] (22.772 ms) : 0, 22772
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (845.798 ms) : 0, 845798
BytebuddyAgent [candidate] (836.564 ms) : 0, 836564
GlobalTracer [baseline] (230.67 ms) : 0, 230670
GlobalTracer [candidate] (228.441 ms) : 0, 228441
AppSec [baseline] (56.186 ms) : 0, 56186
AppSec [candidate] (56.029 ms) : 0, 56029
Debugger [baseline] (4.188 ms) : 0, 4188
Debugger [candidate] (4.185 ms) : 0, 4185
Remote Config [baseline] (606.666 µs) : 0, 607
Remote Config [candidate] (608.4 µs) : 0, 608
Telemetry [baseline] (8.938 ms) : 0, 8938
Telemetry [candidate] (8.824 ms) : 0, 8824
IAST [baseline] (22.873 ms) : 0, 22873
IAST [candidate] (22.771 ms) : 0, 22771
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (840.31 ms) : 0, 840310
BytebuddyAgent [candidate] (832.673 ms) : 0, 832673
GlobalTracer [baseline] (229.303 ms) : 0, 229303
GlobalTracer [candidate] (228.36 ms) : 0, 228360
AppSec [baseline] (56.698 ms) : 0, 56698
AppSec [candidate] (56.183 ms) : 0, 56183
Debugger [baseline] (4.157 ms) : 0, 4157
Debugger [candidate] (4.17 ms) : 0, 4170
Remote Config [baseline] (627.165 µs) : 0, 627
Remote Config [candidate] (607.078 µs) : 0, 607
Telemetry [baseline] (8.756 ms) : 0, 8756
Telemetry [candidate] (8.761 ms) : 0, 8761
IAST [baseline] (22.624 ms) : 0, 22624
IAST [candidate] (22.327 ms) : 0, 22327
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056955
Total [baseline] (10.521 s) : 0, 10520857
Agent [candidate] (1.049 s) : 0, 1049085
Total [candidate] (10.463 s) : 0, 10463497
section appsec
Agent [baseline] (1.195 s) : 0, 1194793
Total [baseline] (10.776 s) : 0, 10776338
Agent [candidate] (1.198 s) : 0, 1198257
Total [candidate] (10.761 s) : 0, 10760753
section iast
Agent [baseline] (1.177 s) : 0, 1177122
Total [baseline] (11.011 s) : 0, 11010755
Agent [candidate] (1.186 s) : 0, 1185596
Total [candidate] (11.05 s) : 0, 11050024
section profiling
Agent [baseline] (1.273 s) : 0, 1272904
Total [baseline] (10.867 s) : 0, 10866579
Agent [candidate] (1.277 s) : 0, 1276503
Total [candidate] (10.883 s) : 0, 10883120
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent appsec 1.195 s 137.838 ms (13.0%)
Agent iast 1.177 s 120.167 ms (11.4%)
Agent profiling 1.273 s 215.949 ms (20.4%)
Total tracing 10.521 s -
Total appsec 10.776 s 255.481 ms (2.4%)
Total iast 11.011 s 489.898 ms (4.7%)
Total profiling 10.867 s 345.722 ms (3.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.049 s -
Agent appsec 1.198 s 149.172 ms (14.2%)
Agent iast 1.186 s 136.512 ms (13.0%)
Agent profiling 1.277 s 227.418 ms (21.7%)
Total tracing 10.463 s -
Total appsec 10.761 s 297.256 ms (2.8%)
Total iast 11.05 s 586.527 ms (5.6%)
Total profiling 10.883 s 419.623 ms (4.0%) 
gantt
    title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (721.994 ms) : 0, 721994
BytebuddyAgent [candidate] (717.757 ms) : 0, 717757
GlobalTracer [baseline] (239.751 ms) : 0, 239751
GlobalTracer [candidate] (238.815 ms) : 0, 238815
AppSec [baseline] (55.018 ms) : 0, 55018
AppSec [candidate] (54.831 ms) : 0, 54831
Debugger [baseline] (4.463 ms) : 0, 4463
Debugger [candidate] (4.46 ms) : 0, 4460
Remote Config [baseline] (703.744 µs) : 0, 704
Remote Config [candidate] (696.795 µs) : 0, 697
Telemetry [baseline] (15.167 ms) : 0, 15167
Telemetry [candidate] (12.805 ms) : 0, 12805
section appsec
BytebuddyAgent [baseline] (739.038 ms) : 0, 739038
BytebuddyAgent [candidate] (740.888 ms) : 0, 740888
GlobalTracer [baseline] (235.501 ms) : 0, 235501
GlobalTracer [candidate] (236.24 ms) : 0, 236240
AppSec [baseline] (176.347 ms) : 0, 176347
AppSec [candidate] (177.017 ms) : 0, 177017
Debugger [baseline] (4.293 ms) : 0, 4293
Debugger [candidate] (4.294 ms) : 0, 4294
Remote Config [baseline] (641.965 µs) : 0, 642
Remote Config [candidate] (648.697 µs) : 0, 649
Telemetry [baseline] (8.214 ms) : 0, 8214
Telemetry [candidate] (8.531 ms) : 0, 8531
IAST [baseline] (22.022 ms) : 0, 22022
IAST [candidate] (21.527 ms) : 0, 21527
section iast
BytebuddyAgent [baseline] (836.498 ms) : 0, 836498
BytebuddyAgent [candidate] (842.495 ms) : 0, 842495
GlobalTracer [baseline] (228.598 ms) : 0, 228598
GlobalTracer [candidate] (229.958 ms) : 0, 229958
AppSec [baseline] (56.002 ms) : 0, 56002
AppSec [candidate] (56.461 ms) : 0, 56461
Debugger [baseline] (4.187 ms) : 0, 4187
Debugger [candidate] (4.239 ms) : 0, 4239
Remote Config [baseline] (612.01 µs) : 0, 612
Remote Config [candidate] (642.978 µs) : 0, 643
Telemetry [baseline] (8.861 ms) : 0, 8861
Telemetry [candidate] (8.835 ms) : 0, 8835
IAST [baseline] (22.669 ms) : 0, 22669
IAST [candidate] (23.163 ms) : 0, 23163
section profiling
BytebuddyAgent [baseline] (709.216 ms) : 0, 709216
BytebuddyAgent [candidate] (710.403 ms) : 0, 710403
GlobalTracer [baseline] (348.145 ms) : 0, 348145
GlobalTracer [candidate] (350.258 ms) : 0, 350258
AppSec [baseline] (54.214 ms) : 0, 54214
AppSec [candidate] (53.561 ms) : 0, 53561
Debugger [baseline] (4.261 ms) : 0, 4261
Debugger [candidate] (4.263 ms) : 0, 4263
Remote Config [baseline] (690.516 µs) : 0, 691
Remote Config [candidate] (705.92 µs) : 0, 706
Telemetry [baseline] (8.893 ms) : 0, 8893
Telemetry [candidate] (8.956 ms) : 0, 8956
ProfilingAgent [baseline] (102.608 ms) : 0, 102608
ProfilingAgent [candidate] (103.376 ms) : 0, 103376
Profiling [baseline] (102.635 ms) : 0, 102635
Profiling [candidate] (103.403 ms) : 0, 103403
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-03-28T15:11:04 2025-03-28T15:18:48
git_branch master smola/retry-flaky-tests
git_commit_date 1743173661 1743173178
git_commit_sha 86e5bec fdf8c7f
release_version 1.48.0-SNAPSHOT~86e5bec219 1.48.0-SNAPSHOT~fdf8c7f65e
start_time 2025-03-28T15:10:50 2025-03-28T15:18:34
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1743175527 1743175527
ci_job_id 870218829 870218829
ci_pipeline_id 60342509 60342509
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-xjcpkyqs-project-304-concurrent-0-9kuajp7e 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-xjcpkyqs-project-304-concurrent-0-9kuajp7e 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 13 metrics, 16 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling worse
[+42.508µs; +94.885µs] or [+2.843%; +6.346%]		 unstable
[-758.800op/s; +284.726op/s] or [-23.713%; +8.898%]		 1.564ms 2962.963op/s 1.495ms 3200.000op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219
    dateFormat X
    axisFormat %s
section baseline
no_agent (382.498 µs) : 363, 402
.   : milestone, 382,
iast (512.562 µs) : 491, 535
.   : milestone, 513,
iast_FULL (730.742 µs) : 709, 753
.   : milestone, 731,
iast_GLOBAL (563.928 µs) : 541, 587
.   : milestone, 564,
iast_HARDCODED_SECRET_DISABLED (511.166 µs) : 489, 533
.   : milestone, 511,
iast_INACTIVE (469.425 µs) : 448, 491
.   : milestone, 469,
iast_TELEMETRY_OFF (499.325 µs) : 478, 521
.   : milestone, 499,
tracing (460.804 µs) : 440, 481
.   : milestone, 461,
section candidate
no_agent (379.757 µs) : 359, 400
.   : milestone, 380,
iast (518.475 µs) : 496, 541
.   : milestone, 518,
iast_FULL (731.173 µs) : 709, 753
.   : milestone, 731,
iast_GLOBAL (567.882 µs) : 544, 591
.   : milestone, 568,
iast_HARDCODED_SECRET_DISABLED (517.748 µs) : 496, 540
.   : milestone, 518,
iast_INACTIVE (461.513 µs) : 441, 482
.   : milestone, 462,
iast_TELEMETRY_OFF (498.083 µs) : 476, 520
.   : milestone, 498,
tracing (464.242 µs) : 443, 486
.   : milestone, 464,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 382.498 µs [362.909 µs, 402.088 µs] -
iast 512.562 µs [490.573 µs, 534.551 µs] 130.064 µs (34.0%)
iast_FULL 730.742 µs [708.661 µs, 752.823 µs] 348.244 µs (91.0%)
iast_GLOBAL 563.928 µs [541.271 µs, 586.585 µs] 181.429 µs (47.4%)
iast_HARDCODED_SECRET_DISABLED 511.166 µs [489.116 µs, 533.216 µs] 128.668 µs (33.6%)
iast_INACTIVE 469.425 µs [448.029 µs, 490.821 µs] 86.926 µs (22.7%)
iast_TELEMETRY_OFF 499.325 µs [477.557 µs, 521.093 µs] 116.827 µs (30.5%)
tracing 460.804 µs [440.201 µs, 481.406 µs] 78.306 µs (20.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 379.757 µs [359.49 µs, 400.025 µs] -
iast 518.475 µs [496.308 µs, 540.641 µs] 138.717 µs (36.5%)
iast_FULL 731.173 µs [709.219 µs, 753.127 µs] 351.416 µs (92.5%)
iast_GLOBAL 567.882 µs [544.389 µs, 591.374 µs] 188.124 µs (49.5%)
iast_HARDCODED_SECRET_DISABLED 517.748 µs [495.964 µs, 539.531 µs] 137.99 µs (36.3%)
iast_INACTIVE 461.513 µs [440.569 µs, 482.457 µs] 81.755 µs (21.5%)
iast_TELEMETRY_OFF 498.083 µs [476.436 µs, 519.729 µs] 118.325 µs (31.2%)
tracing 464.242 µs [442.911 µs, 485.573 µs] 84.485 µs (22.2%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.362 ms) : 1341, 1382
.   : milestone, 1362,
appsec (1.742 ms) : 1718, 1766
.   : milestone, 1742,
appsec_no_iast (1.723 ms) : 1699, 1747
.   : milestone, 1723,
code_origins (1.693 ms) : 1666, 1720
.   : milestone, 1693,
iast (1.525 ms) : 1501, 1550
.   : milestone, 1525,
profiling (1.495 ms) : 1471, 1519
.   : milestone, 1495,
tracing (1.504 ms) : 1479, 1528
.   : milestone, 1504,
section candidate
no_agent (1.362 ms) : 1341, 1382
.   : milestone, 1362,
appsec (1.736 ms) : 1711, 1760
.   : milestone, 1736,
appsec_no_iast (1.721 ms) : 1697, 1745
.   : milestone, 1721,
code_origins (1.665 ms) : 1638, 1693
.   : milestone, 1665,
iast (1.527 ms) : 1503, 1552
.   : milestone, 1527,
profiling (1.564 ms) : 1539, 1588
.   : milestone, 1564,
tracing (1.502 ms) : 1478, 1526
.   : milestone, 1502,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.341 ms, 1.382 ms] -
appsec 1.742 ms [1.718 ms, 1.766 ms] 380.516 µs (27.9%)
appsec_no_iast 1.723 ms [1.699 ms, 1.747 ms] 361.681 µs (26.6%)
code_origins 1.693 ms [1.666 ms, 1.72 ms] 331.027 µs (24.3%)
iast 1.525 ms [1.501 ms, 1.55 ms] 163.505 µs (12.0%)
profiling 1.495 ms [1.471 ms, 1.519 ms] 133.616 µs (9.8%)
tracing 1.504 ms [1.479 ms, 1.528 ms] 142.273 µs (10.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.341 ms, 1.382 ms] -
appsec 1.736 ms [1.711 ms, 1.76 ms] 374.015 µs (27.5%)
appsec_no_iast 1.721 ms [1.697 ms, 1.745 ms] 359.003 µs (26.4%)
code_origins 1.665 ms [1.638 ms, 1.693 ms] 303.585 µs (22.3%)
iast 1.527 ms [1.503 ms, 1.552 ms] 165.611 µs (12.2%)
profiling 1.564 ms [1.539 ms, 1.588 ms] 202.101 µs (14.8%)
tracing 1.502 ms [1.478 ms, 1.526 ms] 140.17 µs (10.3%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/retry-flaky-tests
git_commit_date 1743173661 1743173178
git_commit_sha 86e5bec fdf8c7f
release_version 1.48.0-SNAPSHOT~86e5bec219 1.48.0-SNAPSHOT~fdf8c7f65e
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1743175989 1743175989
ci_job_id 870218832 870218832
ci_pipeline_id 60342509 60342509
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-vapngg-f-project-304-concurrent-0-ys24bk0a 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-vapngg-f-project-304-concurrent-0-ys24bk0a 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.34 ms) : 2296, 2383
.   : milestone, 2340,
iast (2.114 ms) : 2058, 2169
.   : milestone, 2114,
iast_GLOBAL (2.151 ms) : 2096, 2207
.   : milestone, 2151,
profiling (1.962 ms) : 1919, 2006
.   : milestone, 1962,
tracing (1.948 ms) : 1905, 1991
.   : milestone, 1948,
section candidate
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.337 ms) : 2294, 2381
.   : milestone, 2337,
iast (2.108 ms) : 2053, 2164
.   : milestone, 2108,
iast_GLOBAL (2.158 ms) : 2103, 2214
.   : milestone, 2158,
profiling (2.003 ms) : 1956, 2050
.   : milestone, 2003,
tracing (1.946 ms) : 1904, 1988
.   : milestone, 1946,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.47 ms [1.459 ms, 1.482 ms] -
appsec 2.34 ms [2.296 ms, 2.383 ms] 869.247 µs (59.1%)
iast 2.114 ms [2.058 ms, 2.169 ms] 643.439 µs (43.8%)
iast_GLOBAL 2.151 ms [2.096 ms, 2.207 ms] 681.053 µs (46.3%)
profiling 1.962 ms [1.919 ms, 2.006 ms] 491.811 µs (33.4%)
tracing 1.948 ms [1.905 ms, 1.991 ms] 477.591 µs (32.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.461 ms, 1.484 ms] -
appsec 2.337 ms [2.294 ms, 2.381 ms] 864.662 µs (58.7%)
iast 2.108 ms [2.053 ms, 2.164 ms] 635.897 µs (43.2%)
iast_GLOBAL 2.158 ms [2.103 ms, 2.214 ms] 686.017 µs (46.6%)
profiling 2.003 ms [1.956 ms, 2.05 ms] 530.735 µs (36.0%)
tracing 1.946 ms [1.904 ms, 1.988 ms] 473.567 µs (32.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~fdf8c7f65e, baseline=1.48.0-SNAPSHOT~86e5bec219
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.985 s) : 14985000, 14985000
.   : milestone, 14985000,
appsec (15.261 s) : 15261000, 15261000
.   : milestone, 15261000,
iast (19.004 s) : 19004000, 19004000
.   : milestone, 19004000,
iast_GLOBAL (17.984 s) : 17984000, 17984000
.   : milestone, 17984000,
profiling (15.241 s) : 15241000, 15241000
.   : milestone, 15241000,
tracing (15.157 s) : 15157000, 15157000
.   : milestone, 15157000,
section candidate
no_agent (15.654 s) : 15654000, 15654000
.   : milestone, 15654000,
appsec (14.792 s) : 14792000, 14792000
.   : milestone, 14792000,
iast (18.809 s) : 18809000, 18809000
.   : milestone, 18809000,
iast_GLOBAL (18.179 s) : 18179000, 18179000
.   : milestone, 18179000,
profiling (14.979 s) : 14979000, 14979000
.   : milestone, 14979000,
tracing (15.266 s) : 15266000, 15266000
.   : milestone, 15266000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.985 s [14.985 s, 14.985 s] -
appsec 15.261 s [15.261 s, 15.261 s] 276.0 ms (1.8%)
iast 19.004 s [19.004 s, 19.004 s] 4.019 s (26.8%)
iast_GLOBAL 17.984 s [17.984 s, 17.984 s] 2.999 s (20.0%)
profiling 15.241 s [15.241 s, 15.241 s] 256.0 ms (1.7%)
tracing 15.157 s [15.157 s, 15.157 s] 172.0 ms (1.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.654 s [15.654 s, 15.654 s] -
appsec 14.792 s [14.792 s, 14.792 s] -862.0 ms (-5.5%)
iast 18.809 s [18.809 s, 18.809 s] 3.155 s (20.2%)
iast_GLOBAL 18.179 s [18.179 s, 18.179 s] 2.525 s (16.1%)
profiling 14.979 s [14.979 s, 14.979 s] -675.0 ms (-4.3%)
tracing 15.266 s [15.266 s, 15.266 s] -388.0 ms (-2.5%)

@smola smola force-pushed the smola/retry-flaky-tests branch 3 times, most recently from b55b57c to 17f6dc7 Compare December 20, 2024 14:02
@smola smola force-pushed the smola/retry-flaky-tests branch from bb572fc to 0288010 Compare March 17, 2025 16:00
@smola smola force-pushed the smola/retry-flaky-tests branch from 97ab98f to 33bf90b Compare March 28, 2025 08:16
@pr-commenter
Copy link

pr-commenter bot commented Mar 28, 2025 •
edited
Loading

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/retry-flaky-tests
git_commit_date 1743167893 1743173178
git_commit_sha 4f278de fdf8c7f
See matching parameters
Baseline Candidate
ci_job_date 1743175064 1743175064
ci_job_id 870218840 870218840
ci_pipeline_id 60342509 60342509
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Mar 28, 2025 •
edited
Loading

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/retry-flaky-tests
git_commit_date 1743167893 1743173178
git_commit_sha 4f278de fdf8c7f
See matching parameters
Baseline Candidate
ci_job_date 1743175099 1743175099
ci_job_id 870218844 870218844
ci_pipeline_id 60342509 60342509
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

@smola smola force-pushed the smola/retry-flaky-tests branch from 33bf90b to 77e94eb Compare March 28, 2025 10:56
@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Mar 28, 2025
@smola smola force-pushed the smola/retry-flaky-tests branch from 34c7a1d to fdf8c7f Compare March 28, 2025 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@smola