Allow simple tracing for users that do not use more than one handler per incoming message

[bbrandt]

Describe the feature.

Is your feature related to a problem? Please describe.

For a simple message handler that does not send any messages, we have 2 spans. One for "processing" of the message and another representing the handler being executed.

• process message (id:1)
  • MyHandler (id:2, parent_id:1)

Each of these spans have different tags with these sources contributing to tags:

• process message
  • NSB message headers and things from StartIncomingPipelineActivity(...)
  • Tags added from a custom behavior of IInvokeHandlerContext
• MyHandler
  • HandlerType full name and saga id in StartHandlerActivity(...)
  • Activity.Current?.AddTag(...) within the handler

If you are piecing together this data using a query engine of some sort, like Azure Monitor's KQL, for example, then there are a few issues that get cumbersome to deal with.

1. If you tag a business specific property using Activity.Current?.AddTag(...) it goes on the "MyHandler" span, but if you try to do something clever with an IInvokeHandlerContext behavior it goes on the "process message" span even if you try to add your tags after the call to next() in your behavior. Until Provide the ability to capture message bodies as tags in the OpenTelemetry span (activity) for the incoming message #6729 is implemented, we are trying to implement our own cross-cutting layer for tagging key/value of certain message properties, but doing this in a behavior always places it on the "process message" span.

2. The "process message" span is an extra "join" away from our log messages attached to the "MyHandler" span. So, if we are querying, we'd have to get all "process message" spans matching the message type and business specific property we care about and then we'd join to the "MyHandler" spans to get all those span ids we care about. From there we'd have to join to logs and exceptions to just see a flat view of the flow through a single span. In a tool like Azure Monitor, with ~600GB/day accumulating, having this extra join can make things slow.
   Before OpenTelemetry, the teams I work with are used to having only logs (app insights calls them "traces"), with NSB behaviors that write a log at the start of handler processing as well as message sending. These special messages hold business context and are used as locators, like this:

let pIds = (traces
| where message has "Handling MyMessage; OrderId=12345"
| project operation_ParentId);
traces
| where operation_ParentId in (pIds)
| order by timestamp asc;

With NSB Otel, we'd like to simply change this to:

let pIds = (requests
| where name has "MyMessage" and  customDimensions["OrderId"]==12345
| project operation_ParentId);
traces
| where operation_ParentId in (pIds)
| order by timestamp asc;

3. It seems like it may be costing us more having both the "process message" and "MyHandler" spans, but we are getting no value from it, because we never have more than one handler per incoming message per service.

Describe the requested feature

Allow flattening of "process message" and "MyHandler" into a single span. In other words, allow us to disable creation of the span in ActivityFactory.StartHandlerActivity(...) and corresponding disposing of the handler span. Set the DisplayName and other properties that were previously set on "MyHandler" instead on the "process message" span.

Describe alternatives you've considered

1. Allow users to replace the implementation of IActivityFactory and make all other types that ActivityFactory access public.
2. Run all telemetry through a Otel Collector rule that flattens these 2 spans. (Overly complex setup.)
3. Implement an OpenTelemetry processor to flatten these 2 spans into a single span.

Additional Context

No response

[jpalac]

Thank you for submitting the feature request - we will consider it in a future platform enhancement release.