loki-enhance-middleware

loki-enhance-middleware hijacks log push requests sent to loki and modifies it.

Deploy

docker-compose.yaml

services:
  loki:
    image: grafana/loki:2.6.1
    restart: always
    expose:
      - "3100"
    # ...

  enhance_middleware:
      image: nihiue/loki_enhance_middleware:latest
      restart: always

      environment:
        - LOKI_HOST=http://loki:3100
        - WORKER_COUNT=1
        - PORT=3100

      expose:
        - 3100

Config LogAgent

LogAgent needs to:

• Send requests to middleware
• Add placeholder to log line for middle to process, see Module - Geo IP for example

promtail-config.yaml

clients:
  - url: http://enhance_middleware:3100/loki/api/v1/push

Module - Geo IP

Injects GeoIP info for any log source

Powered by maxmind and maxmind-npm

promtail-config.yaml

scrape_configs:
  - job_name: caddy
    pipeline_stages:
      - json:
          expressions:
            level:
            status:
            host: request.host
            method: request.method
            url: request.uri
            remote_addr: request.remote_addr

      - labels:
          level:
          status:
          method:

      - template:
          source: output_msg
          template: 'url="{{ .url }}" host="{{ .host }}" GeoIP_Source="{{.remote_addr }}"'

      - output:
          source: output_msg

GeoIP_Source="[IP]" is the placeholder, and it will be replaced by geo-ip fileds.

Log line

foo=bar GeoIP_Source="22.22.22.22" abc=xyz

Result

foo=bar geo_ip_asn="HostSlick" geo_ip_continent="North America" geo_ip_city="Ashburn" geo_ip_city_geoname_id="4744870" geo_ip_country="United States" geo_ip_country_geoname_id="6252001" geo_ip_country_iso_code="US" geo_ip_latitude="39.018" geo_ip_longitude="-77.539" abc=xyz

Module - UserAgent Detect

Parse user_agent field to structure data.

Powered by device-detector-js

Placeholder: Device_UA_Source="[UA]"

Log line - Normal

Device_UA_Source="Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"

Result

ua_client="Chrome Mobile;90.0" ua_device="Google;Pixel 5" ua_os="Android;11.0"

Log line - Search Engine Bot

Device_UA_Source="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Result

ua_bot="Googlebot"