Inject trace context into AWS Step Functions input #7585
+352
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DylanLovesCoffee
added
type: feature request
inst: aws sdk
amarziali
requested changes
Sep 9, 2024
...s-java-sdk-2.2/src/main/java/datadog/trace/instrumentation/aws/v2/AwsSdkClientDecorator.java
Outdated
Show resolved
Hide resolved
...s-java-sdk-2.2/src/main/java/datadog/trace/instrumentation/aws/v2/AwsSdkClientDecorator.java
Outdated
Show resolved
Hide resolved
...s-java-sdk-2.2/src/main/java/datadog/trace/instrumentation/aws/v2/AwsSdkClientDecorator.java
Outdated
Show resolved
Hide resolved
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1048920
Total [baseline] (8.661 s) : 0, 8661429
Agent [candidate] (1.043 s) : 0, 1042780
Total [candidate] (8.618 s) : 0, 8617655
section iast
Agent [baseline] (1.17 s) : 0, 1170402
Total [baseline] (9.178 s) : 0, 9178009
Agent [candidate] (1.172 s) : 0, 1171972
Total [candidate] (9.204 s) : 0, 9204182
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.179 s) : 0, 1178837
Total [baseline] (9.191 s) : 0, 9191247
Agent [candidate] (1.173 s) : 0, 1173240
Total [candidate] (9.204 s) : 0, 9204021
section iast_TELEMETRY_OFF
Agent [baseline] (1.168 s) : 0, 1167522
Total [baseline] (9.158 s) : 0, 9158343
Agent [candidate] (1.172 s) : 0, 1172299
Total [candidate] (9.228 s) : 0, 9227990
gantt
title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.861 ms) : 0, 719861
BytebuddyAgent [candidate] (717.792 ms) : 0, 717792
GlobalTracer [baseline] (242.997 ms) : 0, 242997
GlobalTracer [candidate] (242.564 ms) : 0, 242564
AppSec [baseline] (55.108 ms) : 0, 55108
AppSec [candidate] (55.071 ms) : 0, 55071
Remote Config [baseline] (714.36 µs) : 0, 714
Remote Config [candidate] (708.365 µs) : 0, 708
Telemetry [baseline] (15.038 ms) : 0, 15038
Telemetry [candidate] (11.448 ms) : 0, 11448
section iast
BytebuddyAgent [baseline] (833.784 ms) : 0, 833784
BytebuddyAgent [candidate] (834.465 ms) : 0, 834465
GlobalTracer [baseline] (232.888 ms) : 0, 232888
GlobalTracer [candidate] (232.798 ms) : 0, 232798
IAST [baseline] (22.634 ms) : 0, 22634
IAST [candidate] (22.966 ms) : 0, 22966
AppSec [baseline] (56.64 ms) : 0, 56640
AppSec [candidate] (57.202 ms) : 0, 57202
Remote Config [baseline] (615.226 µs) : 0, 615
Remote Config [candidate] (622.302 µs) : 0, 622
Telemetry [baseline] (8.683 ms) : 0, 8683
Telemetry [candidate] (8.685 ms) : 0, 8685
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (840.572 ms) : 0, 840572
BytebuddyAgent [candidate] (835.138 ms) : 0, 835138
GlobalTracer [baseline] (233.65 ms) : 0, 233650
GlobalTracer [candidate] (233.638 ms) : 0, 233638
IAST [baseline] (22.853 ms) : 0, 22853
IAST [candidate] (22.716 ms) : 0, 22716
AppSec [baseline] (57.103 ms) : 0, 57103
AppSec [candidate] (57.215 ms) : 0, 57215
Remote Config [baseline] (630.236 µs) : 0, 630
Remote Config [candidate] (602.418 µs) : 0, 602
Telemetry [baseline] (8.667 ms) : 0, 8667
Telemetry [candidate] (8.67 ms) : 0, 8670
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (831.53 ms) : 0, 831530
BytebuddyAgent [candidate] (833.037 ms) : 0, 833037
GlobalTracer [baseline] (232.786 ms) : 0, 232786
GlobalTracer [candidate] (234.647 ms) : 0, 234647
IAST [baseline] (27.852 ms) : 0, 27852
IAST [candidate] (26.566 ms) : 0, 26566
AppSec [baseline] (50.968 ms) : 0, 50968
AppSec [candidate] (53.418 ms) : 0, 53418
Remote Config [baseline] (606.679 µs) : 0, 607
Remote Config [candidate] (623.114 µs) : 0, 623
Telemetry [baseline] (8.523 ms) : 0, 8523
Telemetry [candidate] (8.814 ms) : 0, 8814
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.041 s) : 0, 1040752
Total [baseline] (10.52 s) : 0, 10520235
Agent [candidate] (1.05 s) : 0, 1050414
Total [candidate] (10.431 s) : 0, 10431401
section appsec
Agent [baseline] (1.19 s) : 0, 1189566
Total [baseline] (10.757 s) : 0, 10756882
Agent [candidate] (1.187 s) : 0, 1186514
Total [candidate] (10.729 s) : 0, 10728855
section iast
Agent [baseline] (1.173 s) : 0, 1173029
Total [baseline] (10.954 s) : 0, 10953656
Agent [candidate] (1.174 s) : 0, 1174109
Total [candidate] (10.917 s) : 0, 10916913
section profiling
Agent [baseline] (1.277 s) : 0, 1277447
Total [baseline] (10.937 s) : 0, 10936740
Agent [candidate] (1.264 s) : 0, 1264037
Total [candidate] (10.834 s) : 0, 10833502
gantt
title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.81 ms) : 0, 715810
BytebuddyAgent [candidate] (721.659 ms) : 0, 721659
GlobalTracer [baseline] (242.6 ms) : 0, 242600
GlobalTracer [candidate] (244.764 ms) : 0, 244764
AppSec [baseline] (54.953 ms) : 0, 54953
AppSec [candidate] (55.587 ms) : 0, 55587
Remote Config [baseline] (713.725 µs) : 0, 714
Remote Config [candidate] (730.891 µs) : 0, 731
Telemetry [baseline] (11.46 ms) : 0, 11460
Telemetry [candidate] (12.349 ms) : 0, 12349
section appsec
BytebuddyAgent [baseline] (737.224 ms) : 0, 737224
BytebuddyAgent [candidate] (735.167 ms) : 0, 735167
GlobalTracer [baseline] (240.788 ms) : 0, 240788
GlobalTracer [candidate] (239.964 ms) : 0, 239964
AppSec [baseline] (176.518 ms) : 0, 176518
AppSec [candidate] (176.061 ms) : 0, 176061
Remote Config [baseline] (659.183 µs) : 0, 659
Remote Config [candidate] (672.633 µs) : 0, 673
Telemetry [baseline] (8.315 ms) : 0, 8315
Telemetry [candidate] (8.628 ms) : 0, 8628
IAST [baseline] (21.696 ms) : 0, 21696
IAST [candidate] (21.587 ms) : 0, 21587
section iast
BytebuddyAgent [baseline] (835.56 ms) : 0, 835560
BytebuddyAgent [candidate] (835.824 ms) : 0, 835824
GlobalTracer [baseline] (233.287 ms) : 0, 233287
GlobalTracer [candidate] (233.799 ms) : 0, 233799
AppSec [baseline] (56.84 ms) : 0, 56840
AppSec [candidate] (57.197 ms) : 0, 57197
Remote Config [baseline] (614.043 µs) : 0, 614
Remote Config [candidate] (613.089 µs) : 0, 613
Telemetry [baseline] (8.63 ms) : 0, 8630
Telemetry [candidate] (8.644 ms) : 0, 8644
IAST [baseline] (22.917 ms) : 0, 22917
IAST [candidate] (22.844 ms) : 0, 22844
section profiling
ProfilingAgent [baseline] (97.173 ms) : 0, 97173
ProfilingAgent [candidate] (95.488 ms) : 0, 95488
BytebuddyAgent [baseline] (715.577 ms) : 0, 715577
BytebuddyAgent [candidate] (707.205 ms) : 0, 707205
GlobalTracer [baseline] (356.261 ms) : 0, 356261
GlobalTracer [candidate] (354.723 ms) : 0, 354723
AppSec [baseline] (55.924 ms) : 0, 55924
AppSec [candidate] (54.807 ms) : 0, 54807
Remote Config [baseline] (714.282 µs) : 0, 714
Remote Config [candidate] (703.883 µs) : 0, 704
Telemetry [baseline] (9.091 ms) : 0, 9091
Telemetry [candidate] (8.902 ms) : 0, 8902
Profiling [baseline] (97.197 ms) : 0, 97197
Profiling [candidate] (95.514 ms) : 0, 95514
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section baseline
no_agent (1.344 ms) : 1324, 1364
. : milestone, 1344,
appsec (1.735 ms) : 1712, 1759
. : milestone, 1735,
appsec_no_iast (1.758 ms) : 1735, 1781
. : milestone, 1758,
iast (1.516 ms) : 1492, 1540
. : milestone, 1516,
profiling (1.509 ms) : 1485, 1533
. : milestone, 1509,
tracing (1.465 ms) : 1440, 1491
. : milestone, 1465,
section candidate
no_agent (1.343 ms) : 1323, 1364
. : milestone, 1343,
appsec (1.739 ms) : 1715, 1764
. : milestone, 1739,
appsec_no_iast (1.766 ms) : 1743, 1789
. : milestone, 1766,
iast (1.507 ms) : 1482, 1532
. : milestone, 1507,
profiling (1.501 ms) : 1477, 1525
. : milestone, 1501,
tracing (1.501 ms) : 1476, 1526
. : milestone, 1501,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section baseline
no_agent (376.643 µs) : 357, 397
. : milestone, 377,
iast (503.408 µs) : 481, 525
. : milestone, 503,
iast_FULL (745.806 µs) : 724, 768
. : milestone, 746,
iast_GLOBAL (553.047 µs) : 531, 575
. : milestone, 553,
iast_HARDCODED_SECRET_DISABLED (510.675 µs) : 488, 533
. : milestone, 511,
iast_INACTIVE (455.246 µs) : 434, 477
. : milestone, 455,
iast_TELEMETRY_OFF (501.212 µs) : 477, 525
. : milestone, 501,
tracing (448.704 µs) : 428, 470
. : milestone, 449,
section candidate
no_agent (376.817 µs) : 356, 397
. : milestone, 377,
iast (503.031 µs) : 481, 525
. : milestone, 503,
iast_FULL (738.454 µs) : 716, 761
. : milestone, 738,
iast_GLOBAL (552.094 µs) : 530, 574
. : milestone, 552,
iast_HARDCODED_SECRET_DISABLED (504.225 µs) : 483, 526
. : milestone, 504,
iast_INACTIVE (459.268 µs) : 437, 481
. : milestone, 459,
iast_TELEMETRY_OFF (503.023 µs) : 480, 526
. : milestone, 503,
tracing (460.513 µs) : 439, 482
. : milestone, 461,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section baseline
no_agent (15.218 s) : 15218000, 15218000
. : milestone, 15218000,
appsec (14.841 s) : 14841000, 14841000
. : milestone, 14841000,
iast (18.109 s) : 18109000, 18109000
. : milestone, 18109000,
iast_GLOBAL (17.973 s) : 17973000, 17973000
. : milestone, 17973000,
profiling (14.993 s) : 14993000, 14993000
. : milestone, 14993000,
tracing (15.005 s) : 15005000, 15005000
. : milestone, 15005000,
section candidate
no_agent (15.55 s) : 15550000, 15550000
. : milestone, 15550000,
appsec (15.154 s) : 15154000, 15154000
. : milestone, 15154000,
iast (18.326 s) : 18326000, 18326000
. : milestone, 18326000,
iast_GLOBAL (17.829 s) : 17829000, 17829000
. : milestone, 17829000,
profiling (15.194 s) : 15194000, 15194000
. : milestone, 15194000,
tracing (14.905 s) : 14905000, 14905000
. : milestone, 14905000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~2992508a8d, baseline=1.47.0-SNAPSHOT~e7dd598ab0
dateFormat X
axisFormat %s
section baseline
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (2.366 ms) : 2323, 2409
. : milestone, 2366,
iast (2.113 ms) : 2058, 2168
. : milestone, 2113,
iast_GLOBAL (2.158 ms) : 2103, 2214
. : milestone, 2158,
profiling (1.989 ms) : 1945, 2033
. : milestone, 1989,
tracing (1.947 ms) : 1904, 1989
. : milestone, 1947,
section candidate
no_agent (1.475 ms) : 1464, 1487
. : milestone, 1475,
appsec (2.355 ms) : 2312, 2398
. : milestone, 2355,
iast (2.119 ms) : 2064, 2174
. : milestone, 2119,
iast_GLOBAL (2.161 ms) : 2105, 2216
. : milestone, 2161,
profiling (1.971 ms) : 1928, 2015
. : milestone, 1971,
tracing (1.952 ms) : 1909, 1994
. : milestone, 1952,
|
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Show resolved
Hide resolved
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
| // Include httpclient instrumentation for testing because it is a dependency for aws-sdk. | ||
| testImplementation project(':dd-java-agent:instrumentation:apache-httpclient-4') | ||
| testImplementation project(':dd-java-agent:instrumentation:aws-java-sdk-2.2') | ||
| testImplementation 'software.amazon.awssdk:sfn:2.27.2' |
There was a problem hiding this comment.
It mismatches muzzle requirements. Is it expected?
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
Comment on lines
31
to
36
| String ddTraceContextJSON = InputAttributeInjector.buildTraceContext(span); | ||
| // Inject the trace context into the Step Function input | ||
| StringBuilder modifiedInput = | ||
| InputAttributeInjector.getModifiedInput(request.input(), ddTraceContextJSON); | ||
|
|
||
| return request.toBuilder().input(modifiedInput.toString()).build(); |
There was a problem hiding this comment.
Can be deduplicated using a dedicated method:
SdkRequest injectTraceContext(request, span) {
String traceContext = InputAttributeInjector.buildTraceContext(span);
// Inject the trace context into the Step Function input
String modifiedInput = InputAttributeInjector.getModifiedInput(request.input(), traceContext);
return request.toBuilder().input(modifiedInput).build()
}There was a problem hiding this comment.
Since I had to handle both StartExecutionRequest and StartSyncExecutionRequest param types I ended up doing method overloading for injectTraceContext()
dd-java-agent/instrumentation/aws-java-sfn-2.0/src/test/groovy/SfnClientTest.groovy
Outdated
Show resolved
Hide resolved
dd-java-agent/instrumentation/aws-java-sfn-2.0/src/test/groovy/SfnClientTest.groovy
Show resolved
Hide resolved
dd-java-agent/instrumentation/aws-java-sfn-2.0/src/test/groovy/SfnClientTest.groovy
Outdated
Show resolved
Hide resolved
dd-java-agent/instrumentation/aws-java-sfn-2.0/src/test/groovy/SfnClientTest.groovy
Outdated
Show resolved
Hide resolved
|
@PerfectSlayer I think I covered all of the comments/corrections, but for some reason the CI's muzzle check is continuing to fail :/ It seems like it's mostly timing out, so may be unrelated to my changes? |
dougqh
reviewed
Oct 24, 2024
| public class InputAttributeInjector { | ||
| public static String buildTraceContext(AgentSpan span) { | ||
| // Extract span tags | ||
| StringBuilder spanTagsJSON = new StringBuilder(); |
There was a problem hiding this comment.
This worries me because it is a potential injection vector. I would prefer the use of a JsonBuffer that handles escaping properly.
There was a problem hiding this comment.
Ah I updated this with the JsonBuffer but when I run muzzle locally it fails with
error: cannot find symbol
import datadog.trace.bootstrap.JsonBuffer;
^
symbol: class JsonBuffer
location: package datadog.trace.bootstrap
Any ideas on if I did something very obviously wrong?
There was a problem hiding this comment.
I ran into the same muzzle issue trying to use a JSON parsing library when doing the EventBridge instrumentation. I was basically told we can't use additional libraries in instrumentations and to do the JSON parsing manually with stringbuilder.
There are obviously security concerns with this, so it looks like we're blocked until there is a standardized way to parse JSONs in instrumentations in this repo.
There was a problem hiding this comment.
I just merge #7973 It should provide API to build JSON payload from instrumentations (see JsonWriter and JsonMapper).
dougqh
reviewed
Oct 24, 2024
.../aws-java-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/SfnInterceptor.java
Show resolved
Hide resolved
dougqh
requested changes
Oct 24, 2024
avedmala
reviewed
Nov 20, 2024
...a-sfn-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sfn/InputAttributeInjector.java
Outdated
Show resolved
Hide resolved
|
I'm working on #7973 that might help with the JSON part. |
|
@nhulston is this PR ready to be reviewed ? I see that it does not build. Otherwise it can be put in draft for now |
|
Failures appear to be unrelated |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds an instrumentation for AWS SDK Step Functions. This enables tracing for when a Lambda function invokes a Step Function. Trace context is injected into the Step Function's
StartExecutionRequest/StartSyncExecutionRequest.Inputobject.Example of traced step function in the Serverless org: app
The Logs to Traces Reducer will read the trace context from the Step Function logs and create a span for the Step Function.
Motivation
Continues the work done in Python and NodeJS.
Additional Notes
Screenshots of feature:
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [SVLS-5249](https://datadoghq.atlassian.net/browse/SVLS-5249)