i have seen customers very hesitant to move to using Azure Monitor, so just curious

[bartr]

i have seen customers very hesitant to move to using Azure Monitor, so just curious

[bartr]

In the lab, we deploy Fluent Bit, Prometheus and Grafana. These are "CNCF de-facto standards". Fluent Bit and Grafana work well with Azure Log Analytics.

This repo has an example of how to use Fluent Bit with Azure Log Analytics - https://aka.ms/fbla

You can also look at the deploy/fluentbit folder

I did a presentation at Spark on this as well.

It works really well, even in "CNCF shops".

[larryclaman]

Q about the above: a common customer objection to using Azure Monitor as the sink for container logs is that container logs can often be very chatty, generating tons of log data, which leads to higher than desired costs for Azure monitor and the associated log analytics workspaces. We've heard that log filtering for container logs is "coming", but I don't think it's on the near horizon.

To solve this, would a valid pattern be to use Fluent Bit in the middle & leverage its filtering capabilities; eg Container Logs -> Fluent Bit - > [Filter] -> Log Analytics Workspace ? If so, I'd love to see this scenario added to https://aka.ms/fbla !

[bartr]

If you look at deploy/fluentbit/loga-config.yaml in this repo, you'll see basic filtering by log name.

We could easily add an additional filter to the config. Maybe show an example of only 400+ StatusCodes? Would that help?

    [OUTPUT]
        Name            azure
        Match           kube.var.log.containers.webv*.*
    [OUTPUT]
        Name            azure
        Match           kube.var.log.containers.ngsa*.*

[larryclaman]

Yes, I think an example showing only 400+ codes would be really helpful!

Q: I may be missing something basic, but in the file loga-config.yaml, I see there are three parameters that appear to be passed by substitution, but I can't figure out where these get set, specifically:

    [OUTPUT]
        Name            azure
        Match           kube.var.log.containers.webv*.*
        Customer_ID     ${WorkspaceId}
        Shared_Key      ${SharedKey}
        Log_Type        ${webvLog}

Are ${WorkspaceId}, ${SharedKey}, and ${webvLog} env variables, and if so, how do they get set within the configmap? Or, is it not the configmap at all; rather, is it using this feature of fluentbit? https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/variables
(too many layers of indirection for my brain to handle!)

[bartr]

Log_Type comes from the config map

Customer_ID and Shared_Key come from Kubernetes secrets

fluentbit-pod.yaml

      env:
      - name: WorkspaceId
        valueFrom:
          secretKeyRef:
            name: log-secrets
            key: WorkspaceId

      - name: SharedKey
        valueFrom:
          secretKeyRef:
            name: log-secrets
            key: SharedKey

      - name: ngsaLog
        valueFrom:
          configMapKeyRef:
            name: log-config
            key: ngsaLog

      - name: webvLog
        valueFrom:
          configMapKeyRef:
            name: log-config
            key: webvLog