Fluent bit v1.9.10 - ECS task becomes unhealthy

[bradley-carrion]

We are running the following configuration:

• aws-for-fluent-bit: v2.32.2 (which uses fluent-bit v1.9.10 under the hood)
• awsfirelens
• telegraph
• ruby on rails API
• fluent bit config:

[SERVICE]
    flush     1
    log_level info
    # parsers_file is required to use the parser filter
    parsers_file /fluent-bit/parsers/parsers.conf

# TODO: Exists temporarily while we phase out the old publishing mechanism
[INPUT]
    name http
    tag  api_logs
    listen 0.0.0.0
    port 8888

[INPUT]
    name http
    tag  telegraf_metrics
    listen 0.0.0.0
    port 9999

[FILTER]
    name rewrite_tag
    match *-firelens-*
    rule $log \"TraceToken\"\: adx_event false

[FILTER]
    name parser
    match adx_event
    key_name log
    parser json
    reserve_data true

[OUTPUT]
    name                  azure_blob
    match                 adx_event
    account_name          {account_name}
    shared_key            {key}
    path                  {path}
    blob_type             blockblob
    container_name        {blob_container_name}
    auto_create_container on
    tls                   on

[OUTPUT]
    name                  azure_blob
    match                 api_logs
    account_name           {account_name}
    shared_key            {key}
    path                  {path}
    blob_type             blockblob
    container_name        {blob_container_name}
    auto_create_container on
    tls                   on

[OUTPUT]
    name                  azure_blob
    match                 telegraf_metrics
    account_name          {account_name}
    shared_key            {key}
    path                  {path}
    blob_type             blockblob
    container_name        {blob_container_name}
    auto_create_container on
    tls                   on

[OUTPUT]
    Name                  http
    Match                 *-firelens-*
    Host                  {host}
    Port                  443
    URI                   {endpoint}
    Format                json_lines
    tls                   On
    tls.verify            Off

And once we enabled this at scale, we are seeing high volume of the following errors in cloudwatch which we believe to be the cause for the parent ECS Task becoming unhealthy with CPU for the fluent bit container spiking to 100%.

[error] [http_client] broken connection to {our_storage_account}.blob.core.windows.net:443

The series of events we see play out:

1. The fluent bit container, once this event triggers, is pegged around 1 core of CPU
2. The throughput of logs from the container drops significantly to near zero (different for each output)
3. Many hours later (up to 8) we see the Task instance get marked as unhealthy and recycled (sometimes by the ALB health check and sometimes not)

Acknowledging the many factors in play here, we are curious if anyone else here has a similar configuration and / or maybe had experienced similar issues?

[agup006]

1.9.10 is super old, I would recommend trying a newer version or having an aggregator in the middle.

Essentially get the data out of ECS into a process that can then fire that up to blob. It could be another set of Fluent Instances running on EKS that receive and route to Blob

[bradley-carrion]

Thank you @agup006, we successfully fixed by upgrading to v3.1.4. Opened this for aws-for-fluent-bit to fix aws/aws-for-fluent-bit#850