Allow an endpoint-wide opt in option to continue tracing on receive from a publish

[boblangley]

Describe the feature.

Describe the requested feature

Currently continuing tracing on receive is an opt-in option on a per-publish basis via PublishOptions.

For systems where publish/subscribe conversations are short lived, or resources/costs are not a concern, it may make sense to enable it on an entire endpoint basis.

Not having an endpoint-level opt-in setting is purposeful, as we try to be "safe by default". Enabling this on an endpoint level could become costly and/or break sampling.

We would like to hear from users that would like this option added in a future feature release.

If you are interested in this type of option, please let us know by commenting on this issue. To help us make any future feature of this sort work best, please give us some context on why this would be helpful for you. How are you using publish/subscribe traces in your current system? How long running are your traces? Are you using sampling? Any other context would be helpful.

Additional Context

No response

[unixunion]

As I mentioned in support outreach. I want to enable tracing for an existing system, with many teams and many publish and "send" endpoints. And it would be significantly more painful to onboard if I need all developers to "opt-in" for each publish. It seems that "send" by default connects traces.

But an event-driven architecture on pubsub would not connect to the traceparent without this PublisherOptions set for each time the developer calls publish.

It would be better for us if we could blanket enable tracing via publish, and have a option to negate that in the PublisherOptions.

I looked at doing a pipeline for this to force it to trace, and that works but there is no way to opt-out on a per-message level as the pipeline takes precedence.

Here is the workaround, but like I said, this is force-on with no opt-out per-message, but you can opt-out per contract ( message class )

endpointConfiguration.RegisterComponents(components =>
{
    var rulesProvider  = new ConfigurableOpenTelemetryTracingRulesProvider(defaultShouldTrace: true);
    rulesProvider.AddRule(typeof(InternalTest1), false); // suppress tracing for msg class
    rulesProvider.AddRule(typeof(InternalTest2), false); // suppress tracing for msg class
    components.AddSingleton<IOpenTelemetryTracingRulesProvider>(rulesProvider );
}); 
endpointConfiguration.Pipeline.Register(
    stepId: "ApplyOpenTelemetryBehavior",
    behavior: typeof(ApplyOpenTelemetryBehavior),
    description: "Adds OpenTelemetry tracing context to all outgoing messages."
);

The provider

public class ConfigurableOpenTelemetryTracingRulesProvider : IOpenTelemetryTracingRulesProvider
{
    private readonly Dictionary<Type, bool> _rules = new();
    private readonly bool _defaultShouldTrace;

    public ConfigurableOpenTelemetryTracingRulesProvider(bool defaultShouldTrace = true)
    {
        _defaultShouldTrace = defaultShouldTrace;
    }

    public void AddRule(Type messageType, bool shouldTrace)
    {
        _rules[messageType] = shouldTrace;
    }

    public bool ShouldTrace(Type messageType) =>
        _rules.TryGetValue(messageType, out var shouldTrace) ? shouldTrace : _defaultShouldTrace;
}

public interface IOpenTelemetryTracingRulesProvider
{
    bool ShouldTrace(Type messageType);
}

And the pipeline

public class ApplyOpenTelemetryBehavior : Behavior<IOutgoingLogicalMessageContext>
{
    private readonly IOpenTelemetryTracingRulesProvider _rulesProvider;

    public ApplyOpenTelemetryBehavior(IOpenTelemetryTracingRulesProvider rulesProvider)
    {
        _rulesProvider = rulesProvider;
    }

    public override Task Invoke(IOutgoingLogicalMessageContext context, Func<Task> next)
    {
        var messageType = context.Message.Instance.GetType();
        var shouldTrace = _rulesProvider.ShouldTrace(messageType);

        if (shouldTrace)
        {
            // Determine whether to continue or start a new trace
            bool continueTrace = Activity.Current != null;
            context.Headers[Headers.StartNewTrace] = (!continueTrace).ToString();
        }

        return next();
    }
}

[ydie22]

We are interested by such an option. Our workflows are short-lived, and we currently use the trace ID to correlate traces and logs. Being forced to navigate span links makes following the whole thing difficult, we don't have proper tooling support for this. We don't do sampling at this stage, this may come in the future. Notice that an identical option for delayed deliveries would also be welcome. This is even more true for delayed retries, which are semantically very tightly bound to the original trace, even if the time span can be long.