Debugging getmail #4344

tilllt · 2025-02-04T16:04:22Z

tilllt
Feb 4, 2025

My Getmail config is not working for some reason, I have no clue how to debug...

Running getmail --rcfile /tmp/docker-mailserver/getmail/test.cf from within the docker results in:

Error: Could not find the getmail configuration directory.  mkdir ~/.config/getmail/ or specify an alternate directory with the --getmaildir option.

Generally speaking the config files are mounted into the container and look fine.

Answered by polarathene

Feb 5, 2025

Is this a bug in docker-mailserver?

No, below is a full working example which functions fine on both v14 and v15 (presently :edge) of DMS.

As mentioned view the appropriate docs page for the image tag you're using due to the breaking change for getmail in DMS v15.

DMS Getmail example

Create the yaml files, or manually combine them, run the commands below to verify.

$ docker compose -f tls.compose.yaml run --rm gen-certs
$ docker compose up -d

# John has no mail in his inbox (Your local DMS instance with getmail)
$ docker compose exec -it dms-getmail doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=0

# Jane has no mail in her inbox (Your remote DMS instance)
…

View full answer

casperklein · 2025-02-04T17:40:54Z

casperklein
Feb 4, 2025
Maintainer

Use either getmail --getmaildir /tmp/docker-mailserver/getmail --rcfile test.cf --dump or debug-getmail, which does the same, but for all config files present in /tmp/docker-mailserver/getmail

3 replies

tilllt Feb 4, 2025
Author

debug-getmail
Error: configuration file /etc/getmailrc.d/getmailrc* does not exist

There is

/etc/getmailrc_general

and an empty

/etc/getmailrc.d:

tilllt Feb 4, 2025
Author

Is this a bug in docker-mailserver? I didn't manually delete or move anything.

casperklein Feb 4, 2025
Maintainer

Did you enable getmail (ENABLE_GETMAIL=1)? Did you restart (docker compose up --force-recreate -d) the container afterwards?

Edit:

~~The files in /tmp/docker-mailserver/getmail are copied to /etc/getmailrc.d on container startup.~~ (oups, that's currently only true for the :edge image).

The getmail service has been revised, but not released yet. I recommend, that you wait for the next release or use the :edge image in the meantime.

You find the latest getmail documentation (edge tag) here.

If you still have problems with the :edge tag, please open a bug report.

polarathene · 2025-02-05T02:48:38Z

polarathene
Feb 5, 2025
Maintainer

Is this a bug in docker-mailserver?

No, below is a full working example which functions fine on both v14 and v15 (presently :edge) of DMS.

As mentioned view the appropriate docs page for the image tag you're using due to the breaking change for getmail in DMS v15.

DMS Getmail example

Create the yaml files, or manually combine them, run the commands below to verify.

$ docker compose -f tls.compose.yaml run --rm gen-certs
$ docker compose up -d

# John has no mail in his inbox (Your local DMS instance with getmail)
$ docker compose exec -it dms-getmail doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=0

# Jane has no mail in her inbox (Your remote DMS instance)
$ docker compose exec -it dms-remote doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=0

# Send the remote DMS a mail internally (The PERMIT_DOCKER ENV allows to send this without authentication):
$ docker compose exec -it dms-remote swaks --server localhost --from [email protected] --to [email protected]

# Jane has mail now:
$ docker compose exec -it dms-remote doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=1

# Wait until the next getmail poll occurs, and check inboxes again.
# Getmail has retrieved Jane's mail, storing a copy at John's inbox:
$ docker compose exec -it dms-fetch doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=1

# As `delete = false` is the default in `/etc/getmailrc_general`,
# the mail is not deleted from Jane's mailbox:
$ docker compose exec -it dms-remote doveadm mailbox status -u [email protected] messages INBOX
INBOX messages=1

# dms.compose.yaml

services:
  dms-getmail:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest # :14.0
    hostname: mail.example.test
    environment:
      ENABLE_GETMAIL: 1
      # We only change this setting to 1 minute for quicker testing:
      GETMAIL_POLL: 1
    # You'd normally use `volumes` here but for simplicity of the example, all config is contained within `compose.yaml`:
    configs:
      - source: dms-accounts-getmail
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: getmail-jane
        #target: /tmp/docker-mailserver/getmail/jane.cf # v15 has breaking change to path
        target: /tmp/docker-mailserver/getmail-jane.cf # v14

  dms-remote:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest # :14.0
    hostname: mail.remote.test
    environment:
      # This is only used for testing as if mail had arrived via port 25 from another MTA,
      # Allows for us send a test mail easily by trusting any mail client run within the container (like `swaks`):
      PERMIT_DOCKER: container
    configs:
      - source: dms-accounts-remote
        target: /tmp/docker-mailserver/postfix-accounts.cf

# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
  # Basic getmail config to retrieve mail from an account at another mail server via IMAP credentials:
  getmail-jane:
    content: |
      [retriever]
      type = SimpleIMAPSSLRetriever
      server = mail.remote.test
      username = [email protected]
      password = secret

      [destination]
      type = MDA_external
      path = /usr/lib/dovecot/deliver
      allow_root_commands = true
      arguments = ("-d","[email protected]")

  # DMS requires an account to complete setup, provide one for each DMS instance:
  # NOTE:
  # - Both accounts are configured with the same password `secret` (SHA512-CRYPT hashed).
  # - To opt-out of Docker Compose variable interpolation, `$` must be escaped as `$$`.
  dms-accounts-getmail:
    content: |
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  dms-accounts-remote:
    content: |
      [email protected]|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

The above will fail in this case since there's no cert configured and the [retriever] config has the type SimpleIMAPSSLRetriever (DMS imap also expects TLS to be established). For testing locally we can add TLS support to both containers like this:

# dms-tls.compose.yaml

volumes:
  custom-certs:
    name: tls-remote-test
    external: true

services:
  dms-getmail:
    configs:
      - source: dms-trust-custom-ca
        target: /tmp/docker-mailserver/user-patches.sh
    volumes:
      - custom-certs:/tmp/custom-certs:ro

  dms-remote:
    environment:
      SSL_TYPE: manual
      SSL_CERT_PATH: /tmp/custom-certs/remote.test/cert.pem
      SSL_KEY_PATH: /tmp/custom-certs/remote.test/key.pem
    volumes:
      - custom-certs:/tmp/custom-certs:ro

# Copy the root CA cert to a location that `update-ca-certificates` command will install it:
# Alternatively use the long syntax for volumes to mount directly by `subpath`,
# But you will still need to run `update-ca-certificates`.
configs:
  dms-trust-custom-ca:
    content: |
      #!/bin/bash
      cp /tmp/custom-certs/ca/cert.pem /usr/local/share/ca-certificates/smallstep-ca.crt
      update-ca-certificates

# compose.yaml

# Alternative to using the CLI 'merge' syntax with `-f`:
# docker compose -f dms.compose.yaml -f dms-tls.compose.yaml up
# https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/
# https://docs.docker.com/reference/compose-file/merge/
# https://docs.docker.com/compose/how-tos/multiple-compose-files/include/
# https://docs.docker.com/reference/compose-file/include/
include:
  - path:
    - dms.compose.yaml
    - dms-tls.compose.yaml

As per the last YAML snippet, you can either specify separate compose files via CLI or in a compose.yaml by leveraging include with the path key to provide an entry with multiple files to merge (without path the two files would conflict instead of merge due to the same service names).

docker compose up will run the example, and if you'd rather a single compose file without manually merging dms.compose.yaml + dms-tls.compose.yaml you can use docker compose config to output the merged config that Compose has processed.

Certificate provision example (run this first)

To compliment the dms-tls.compose.yaml config above, here is how I generated the local cert for SSL_TYPE=manual to use:

# tls.compose.yaml

# Named data volume for sharing with other compose projects:
# NOTE: For production use, your services would volume mount with `subpath` (requires long syntax):
# https://docs.docker.com/reference/compose-file/services/#volumes
volumes:
  custom-certs:
    name: tls-remote-test

services:
  gen-certs:
    image: smallstep/step-ca
    # `smallstep/step-ca` by default runs as non-root (1000:1000),
    # change to the desired UID/GID ownership of certs generated:
    user: root
    # Persist certs externally:
    volumes:
      - custom-certs:/tmp/certs/:rw
    working_dir: /tmp/certs
    # Support for running the custom script below:
    entrypoint: /tmp/generate-certs.sh
    configs:
      - source: generate-certs
        target: /tmp/generate-certs.sh
        # Make script executable:
        mode: 500

configs:
  generate-certs:
    content: |
      #!/usr/bin/env bash

      # Store root CA and leaf certs in separate directories:
      mkdir -p ca remote.test

      # NOTE: Append the `--force` option to both commands below if you want to run this
      # script multiple times, such as with services using `depends_on: [ gen-certs ]`
      step certificate create 'Smallstep Root CA' ca/cert.pem ca/key.pem \
        --profile root-ca --no-password --insecure

      step certificate create 'Smallstep Leaf' remote.test/cert.pem remote.test/key.pem \
        --san 'mail.remote.test' \
        --ca ca/cert.pem --ca-key ca/key.pem \
        --profile leaf --no-password --insecure

This could be saved as say tls.compose.yaml and run with docker compose -f tls.compose.yaml run --rm gen-certs.

Troubleshooting log

In the example above, without dms-tls.compose.yaml, in this case the functionality fails as it can't successfully connect to the IMAP service of dms-remote container due to lack of TLS configuration. The logs aren't exactly obvious about that at a glance, but IIRC getmail is run as a cron service prior to DMS v15.

When cron fails due to an error while performing the task, it'll send an email to the system user root, which then tries to send that the [email protected] account, but no such account is configured by default, nor is the `POSTMASTER_ADDRESS ENV present to alias it - so the log event will just spam the logs about that many times as it tries to alert the sysadmin.

Full log during this event

dms-getmail-1  | 2025-02-04T21:30:01.746919+00:00 mail postfix/pickup[716]: B645F15FC0C: uid=0 from=<root>
dms-getmail-1  | 2025-02-04T21:30:01.767226+00:00 mail postfix/cleanup[833]: B645F15FC0C: message-id=<[email protected]>
dms-getmail-1  | 2025-02-04T21:30:01.769347+00:00 mail opendkim[602]: B645F15FC0C: localhost [127.0.0.1] not internal
dms-getmail-1  | 2025-02-04T21:30:01.769473+00:00 mail opendkim[602]: B645F15FC0C: not authenticated
dms-getmail-1  | 2025-02-04T21:30:01.773801+00:00 mail opendkim[602]: B645F15FC0C: no signature data
dms-getmail-1  | 2025-02-04T21:30:01.777784+00:00 mail postfix/qmgr[717]: B645F15FC0C: from=<[email protected]>, size=654, nrcpt=1 (queue active)
dms-getmail-1  | 2025-02-04T21:30:01.794855+00:00 mail postfix/cleanup[833]: C1DFD15FC0E: message-id=<[email protected]>
dms-getmail-1  | 2025-02-04T21:30:01.800665+00:00 mail postfix/local[835]: B645F15FC0C: to=<[email protected]>, orig_to=<root>, relay=local, delay=0.1, delays=0.08/0.02/0/0.01, dsn=2.0.0, status=sent (forwarded as C1DFD15FC0E)
dms-getmail-1  | 2025-02-04T21:30:01.801682+00:00 mail postfix/qmgr[717]: B645F15FC0C: removed
dms-getmail-1  | 2025-02-04T21:30:01.802090+00:00 mail postfix/qmgr[717]: C1DFD15FC0E: from=<[email protected]>, size=790, nrcpt=1 (queue active)
dms-getmail-1  | 2025-02-04T21:30:01.890176+00:00 mail dovecot: lmtp(837): Connect from local
dms-getmail-1  | 2025-02-04T21:30:01.914898+00:00 mail dovecot: auth: passwd-file([email protected]): unknown user
dms-getmail-1  | 2025-02-04T21:30:01.932523+00:00 mail postfix/lmtp[836]: C1DFD15FC0E: to=<[email protected]>, orig_to=<root>, relay=mail.example.test[/var/run/dovecot/lmtp], delay=0.14, delays=0.01/0.03/0.05/0.04, dsn=5.1.1, status=bounced (host mail.example.test[/var/run/dovecot/lmtp] said: 550 5.1.1 <[email protected]> User doesn't exist: [email protected] (in reply to RCPT TO command))
dms-getmail-1  | 2025-02-04T21:30:01.934144+00:00 mail dovecot: lmtp(837): Disconnect from local: Logged out (state=READY)
dms-getmail-1  | 2025-02-04T21:30:01.937631+00:00 mail postfix/cleanup[833]: E496A15FC13: message-id=<[email protected]>
dms-getmail-1  | 2025-02-04T21:30:01.939567+00:00 mail postfix/bounce[839]: C1DFD15FC0E: sender non-delivery notification: E496A15FC13
dms-getmail-1  | 2025-02-04T21:30:01.940194+00:00 mail postfix/qmgr[717]: E496A15FC13: from=<>, size=2966, nrcpt=1 (queue active)
dms-getmail-1  | 2025-02-04T21:30:01.940830+00:00 mail postfix/qmgr[717]: C1DFD15FC0E: removed

In DMS v15, this cron job is changed to a a supervisord service instead which has a bash script loop the polling task.

The DMS v15 change will thus make the failure even more silent, not sure if @casperklein wants to look into that

docker-mailserver/target/getmail/getmail-service.sh

Line 37 in 0e61f17

getmail --getmaildir "${GETMAIL_DIR}" --rcfile "${RC_FILE}"

With this example we have the following:

$ docker compose -f dms.compose.yaml up --force-recreate -d
$ docker compose -f dms.compose.yaml exec -it dms-getmail bash

$ getmail --getmaildir /var/lib/getmail --rcfile /etc/getmailrc.d/jane
jane: socket error ([Errno 111] Connection refused)

$ echo $?
127

$ cat /var/log/supervisor/getmail.log
jane: socket error ([Errno 111] Connection refused)
jane: socket error ([Errno 111] Connection refused)
jane: socket error ([Errno 111] Connection refused)

So I guess that's fine, the stderr is logged in it's new service log file, the user just needs to check it.

7 replies

tilllt Feb 5, 2025
Author

That's ok, I read that using IDLE is the much more efficient method, but I am fine with 1 minute, that's what I have at the moment.

polarathene Feb 5, 2025
Maintainer

I have to figure out a good way to export my Let's Encrypt certificates from Traefik to be able to use them for docker-mailserver.

Is there an issue with our acme.json support for Traefik? (see our docs, although the Traefik ones are a little outdated, you shouldn't need SSL_DOMAIN ENV)

Just so it's clear, you only need a certificate for the DMS FQDN (hostname), not for any of the mail addresses. So mail.example.com for DMS, but no need for a example.com cert if you have [email protected].

tilllt Feb 6, 2025
Author

Is there an issue with our acme.json support for Traefik? (see our docs, although the Traefik ones are a little outdated, you shouldn't need SSL_DOMAIN ENV)

Ha! I hadnt seen those, much better, thanks again for your great help, also with my initial question on the minimal setup with getmail and docker mailserver!

As for supporting the IDLE mechanism, is that a technical limitation or should i leave a feature request for future versions. Same goes for the dovecot-flatcurve-fts (https://github.com/slusarz/dovecot-fts-flatcurve) since this will be the "built-in" full text search in future dovecot versions?

casperklein Feb 6, 2025
Maintainer

As for supporting the IDLE mechanism, is that a technical limitation or should i leave a feature request for future versions

We are using getmail6, not getmail. It's a fork, that supports python 3, while the original getmail only works with python 2. In getmail6, there is no --idle option available.

tilllt Feb 6, 2025
Author

Thanks for the clarification, weird that its mentioned in the getmail6 docs but maybe they fortgot to edit it out:

If you are using a single getmailrc file with an IMAP server that understands the IDLE extension from RFC 2177, you can use the --idle=MAILBOX option to specify that getmail should wait on the server to notify getmail of new mail in the specified mailbox after getmail is finished retrieving mail.

https://getmail6.org/configuration.html

tilllt · 2025-02-06T16:46:47Z

tilllt
Feb 6, 2025
Author

I found that its helpful to make an override to the default general getmail config to make it more "talkative"

services:
  mailserver:
    configs:
      - source: getmail_general
        target: /tmp/docker-mailserver/getmail/getmailrc_general.cf

configs:
  getmail_general:
    content: |
      # https://getmail6.org/configuration.html#conf-options
      [options]
      verbose = 2
      read_all = false
      delete = false
      delete_after = 30
      max_messages_per_session = 500
      received = false
      delivered_to = false
      #      message_log = /var/log/getmail.log
      message_log_verbose = true
      message_log_syslog = true

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Mailserver

Debugging getmail #4344

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Docker Mailserver

Debugging getmail #4344

tilllt Feb 4, 2025

DMS Getmail example

Replies: 3 comments · 10 replies

casperklein Feb 4, 2025 Maintainer

tilllt Feb 4, 2025 Author

tilllt Feb 4, 2025 Author

casperklein Feb 4, 2025 Maintainer

polarathene Feb 5, 2025 Maintainer

DMS Getmail example

Certificate provision example (run this first)

Troubleshooting log

tilllt Feb 5, 2025 Author

polarathene Feb 5, 2025 Maintainer

tilllt Feb 6, 2025 Author

casperklein Feb 6, 2025 Maintainer

tilllt Feb 6, 2025 Author

tilllt Feb 6, 2025 Author

tilllt
Feb 4, 2025

Replies: 3 comments 10 replies

casperklein
Feb 4, 2025
Maintainer

tilllt Feb 4, 2025
Author

tilllt Feb 4, 2025
Author

casperklein Feb 4, 2025
Maintainer

polarathene
Feb 5, 2025
Maintainer

tilllt Feb 5, 2025
Author

polarathene Feb 5, 2025
Maintainer

tilllt Feb 6, 2025
Author

casperklein Feb 6, 2025
Maintainer

tilllt Feb 6, 2025
Author

tilllt
Feb 6, 2025
Author