Log Level: Flush low priority message to the cloudwatch in case of errors

[zirkelc]

Hi,

I currently log lots of information as INFO or DEBUG to cloudwatch. In order to reduce my cloudwatch bill, I'm thinking about setting the default log level POWERTOOLS_LOG_LEVEL to ERROR. But then, in case of errors I' don't have the useful debug logs available to analyze the error.

I didn't look the the implementation yet, but maybe it would be possible to collect all DEBUG/INFO/... logs which are less than the configured log level, and then in case of an error logged via logger.error() (or any other log message > log level) to flush all collected logs to cloudwatch. That means only critical logs (log level > POWERTOOLS_LOG_LEVEL) would trigger non-critical logs (log level < POWERTOOLS_LOG_LEVEL) to be logged in cloudwatch.

[dreamorosi]

Hi @zirkelc, thank you for opening this discussion.

From what I understand, this sounds quite similar to this existing feature request #519. If this is the case, I'd like to encourage to 👍 there as this helps us understand demand.

--

Regarding the actual feature, I think this makes sense and it's something we have on our roadmap for the medium term. Before working on this we'd like to better understand the impact of buffering logs on memory usage and find a way to decrease the chances of running out of memory. For example, we're unsure if we should buffer based on a certain amount of messages, or based on a certain size of payloads, or both.

Additionally, we're looking into doing something slightly more advanced than what suggested and not just flush what's in the buffer when you call logger.error() but also when there's an unhandled error, i.e.

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger({ logLevel: 'error' });

export const handler = async (event) => {
  logger.logEventIfEnabled(event); // this is an info log

  logger.debug('some other log');

  throw new Error('something bad happened'); // at this point, before the runtime exits we'd log both these logs present in the buffer
}

In order to do this, however, we need to resolve some open points with the AWS Lambda team since today it's not possible to do this without opting into Lambda Extensions.

Either way, depending on how the conversation goes, we might want to implement this in two waves and first enable the behavior you described, which we could do independently.

[zirkelc]

Hi @dreamorosi

thanks for the quick response! Yes, I think the feature request reflects it very well!

Regarding the buffer size: I think this should be configurable to something like a "sliding window" which collect the most recent N logs based on a metric of size or time, and drop older logs.

The possibility to to trigger the flush of logs on throw new Error is probably already possible by a middleware like middy, but of course having it natively supported by Lambda would be even better.

Please let me know if there's something I can do to help you get this feature started! 👍
In the feature request your colleague mentioned you need company names, if that's still the case, please let me know if I should write an email in name of my my own little startup.

Thank you!

[dreamorosi]

Hi @zirkelc, thank you for the reply and for the added suggestions.

The callout about company names is still valid and would be a huge help. No startup or company is too small to make a difference in our backlog prioritisation, so that would be welcome but not required.

[zirkelc]

Hey @dreamorosi

as said via email, I made a very naive implementation of this feature and I'd like to get your feedback on it if you got a chance. Here's the diff in my forked repo: #3178

All log items less than the configured log level will be stored in a context (the name buffer is already used). The context is an object instead of an array, because I'm using the X-ray Trace ID as key to distinguish between logs from different invocations, in case the Lambda environment was re-used. The log item will be stored as formatted string instead of the raw object to avid attributes from being mutated after they have been logged. When a log item higher than the configured log level is logged, the context will be flushed and reset.

On a sidenote, the way it's implemented could actually be done quite easily by sub-classing the Logger and overriding processLogItem and printLog. Unfortunately, these methods are private instead of protected. Maybe it would make sense to set them to protected to enable features like this and others to be easily implemented in userland?

[dreamorosi]

Closing this in favor of #3410 - the proposal moved onto the RFC phase and will be part of the 2025 backlog for Powertools for AWS (estimated ETA Q1).