Question about the OpenTelemetry tracing layer example

[Oliboy50]

@borchero I have a question about the new example opentelemetry-tracing

My question is about the following piece of code:

let runtime = Runtime::new(service_fn(echo)).layer(OpenTelemetryLayer::new(|| {
    // Make sure that the trace is exported before the Lambda runtime is frozen
    tracer_provider.force_flush();
}));
runtime.run().await?;

Ok(())

I was wondering if this would be the same if this was written this way instead:

let runtime = Runtime::new(service_fn(echo)).layer(OpenTelemetryLayer::new());
runtime.run().await?;

// Make sure that the trace is exported before the Lambda runtime is frozen
tracer_provider.force_flush();

Ok(())

And if it's not the same, could you please explain how it differs? 🙏

[calavera]

It's not the same. When you call runtime.run().await?;, the runtime starts an event loop waiting for messages from Lambda to invoke your code. The purpose if the layer is to flush the traces after completing every invocation that the runtime receives. The runtime will call all the layers inside the event loop. The first snippet makes force_flush to be executed after every invocation.

If you put the code after calling runtime.run().await?;, the force_flush method is only invoked when Lambda destroys the sandbox to process the invocations, not after every invocation.

[Oliboy50]

Thanks for the answer 👌

I have an other (less important) question now:

Is it best to:

"force flushing telemetry data after each Lambda invocation"
VS
"leaving the OpenTelemetry layer flush batches of telemetry data every 30 seconds (for instance) and making sure to flush the remaining data before the Lambda program exits"

?

[borchero]

In AWS Lambda, your execution environment is frozen after every invocation. That means that no background processes can run unless your "main" code currently processes an event. Thus, exporting data on any schedule does not work reliably. You might still get lucky to export data from time to time if the execution environment is reused across function invocations -- you will have an unpredictable delay in telemetry being sent though.

In any case, you will likely lose the telemetry of the last (few) function invocations before the execution environment shuts down (typically after ~5 minutes of no function invocations, function updates, ...) if you're not flushing telemetry after every invocation. The reason is that, by default, the execution environment is just shut down at some point without running any code again. That means that if your last function invocation did not flush telemetry directly after the invocation, it will never get the chance to do so again.

---

Side note: if you have a Lambda extension registered (e.g. the Lambda OTEL collector), your execution environment will be awoken for 2 seconds just before it's shut down. If you're lucky, telemetry can still be exported during that time but this is nothing to be relied upon.

[bnusunny]

You can run a background task with an internal extension. Checkout this example.

https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/examples/extension-internal-flush

[Oliboy50]

Thank you very much for your answers, I had no idea about the freeze of the Lambda runtime (is it documented somewhere?).

I'm glad I asked my second question. Thank you again.