baseline_mailrelay¶

The cake function baseline_mailrelay executes the Ansible role-mailrelay-opensmtpd, which installs and configures opensmtpd to relay emails generated by (Web) applications to authenticated SMTP-Servers (like gmail).

It can also relay emails to the matrix chat.

Simplified, the opensmtpd-mailrelay works as follows:

Opensmtpd uses action and match directives in /etc/smtpd.conf to handle mail generated by Webapps, Cronjobs and so on
The mails are sorted by the "MAIL FROM" header (set by the Mail User Agent) using the match directive, which associates the email with an action
The action defines over which external relay server (for example gmail.com) to relay the mail

Information¶

Key	Value
Playbook path	`plays/baseline/mailrelay.yml`
Role	https://git.blunix.com/ansible-roles/role-mailrelay-opensmtpd
Tags	https://git.blunix.com/ansible-roles/role-mailrelay-opensmtpd/-/tags
Defaults	https://git.blunix.com/ansible-roles/role-mailrelay-opensmtpd/-/blob/master/defaults/main.yml

Config file	Description
/etc/mailname	Contains the hostname the server should use during SMTP connections as well as writing into the email headers
/etc/smtpd.conf	Main config file
/etc/smtpd/	Main config directory
/etc/smtpd/tables	Directory for database-like files, like smtp-user and -password information
/etc/smtpd/tables/monitoring_{from,recipients}	Contains database of email addresses to relay to monitoring (matrix chat)
/etc/smtpd/tables/relay_secrets	Contains relay host authentication data (gmail.com login)
/etc/smtpd/filters	Directory for scripts that scan and modify or accept / reject email
/etc/smtp-to-matrix/config.yml	Contains matrix chat authentication data

Example¶

passwords:

pass edit mailrelay_smtp_to_matrix_username
pass edit mailrelay_smtp_to_matrix_password
compile-passwords.py

inventory/group_vars/all.yml:

# role-mailrelay-opensmtpd
# Hostname for this mailrelay - should not be the servers actual hostname
# Commonly just the domain of the company running the server
# Will be written into the header of all outgoing emails
# Has to be an existing mailserver with a MX record
mailrelay_opensmtpd_mailname: mail.example.com

#mailrelay_smtp_to_matrix_username: from password store
#mailrelay_smtp_to_matrix_password: from password store
mailrelay_smtp_to_matrix_server: "https://matrix.org"
mailrelay_smtp_to_matrix_room_id: "!matrix-monitoring-room-id-here:matrix.org"

inventory/group_vars/util_backup.yml:

mailrelay_opensmtpd_monitoring_linux_users:
  - root
  - borgbackup

inventory/group_vars/cus_www_prod_web.yml:

mailrelay_opensmtpd_relays:

    # Alias for this relay
  - alias: myrelay

    # Relay server DNS name or IP
    server: "mail.example.com"

    # mail-from email address
    mail-from: monitoring@example.com

    # Relay server protocol (Documentation: https://man.openbsd.org/smtpd.conf#host)
    protocol: "smtp+tls"

    # Enforce TLS (default: True)
    tls: True

    # Do not verify TLS certificates (default: False)
    tls: True

    # Relay server port
    port: "587"

    # Relay server SMTP login
    login: opensmtpd@example.com

    # Relay server SMTP password
    # This should be in the password-store!
    password: "{{ cus_www_prod_web_email_password }}"

    # Linux of sender addresses to apply this relay for
    from:
      - root
      - www-data
      - opensmtpd@example.com


  # Simple version
  - alias: backup
    server: "mail.beispiel.de"
    protocol: "smtps"
    tls: True
    port: "465"
    login: backup@beispiel.de
    password: superduperhyperextremelysecret
    from:
      - backuppc

Usage¶

Password for encrypted mails in queue¶

The opensmtpd mail queue is encrypted. To interact with it, you need to enter a password. This password is in ansible-cake in the password store:

CAKE master pass show mailrelay_opensmtpd_conf_queue_encryption
otalR1o1RdygQ7Az12N53e1ytrjk7pee

CAKE master cake ssh web-2 root pub

root@cus-www-prod-web-2 ~ # smtpctl show message pmRXuDvpF4892vik
key> <ENTER mailrelay_opensmtpd_conf_queue_encryption PASSWORD HERE>
root@cus-www-prod-web-2 ~ # smtpctl show envelope pmRXuDvpF4892vik

Common commands¶

Show mails in queue

smtpctl show queue

Show number of mails in queue

smtpctl show queue | nl     # Few mails
smtpctl show queue | wc -l  # Lots of mails

Show size of mail queue directory

du -sh /var/spool/smtpd/

Delete specific mail(s) in the queue

smtpctl show queue
smtpctl remove <queue-id> <que-id>

Delete all mails in the queue

smtpctl remove all

Show a specific emails body

smtpctl show <queue-id>
smtpctl show message a2c25ab9d701c1c6

Show a specific emails envelope

smtpctl show <queue-id>
smtpctl show envelope a2c25ab9d701c1c6

Re-attempt delivery of all mails in the queue

smtpctl schedule all

Debugging while sending¶

To debug the path an email takes on the system, use the following commands:

Sending a test email, setting MAIL FROM to "root":

root@server ~ # echo testbody | mail -s testsubject recipient@example.com

Sending a test email, setting MAIL FROM to "grafana":

root@server ~ # echo fo | mail -s testsubject -a "From: grafana" recipient@example.com

Sending a test email, setting MAIL FROM to a user not defined in any "match" statement:

root@server ~ # echo fo | mail -s testsubject -a "From: nonexistinguser" recipient@example.com
mail: cannot send message: Process exited with a non-zero status

Showing the email in the mail queue:

smtpctl show queue
3ac1dk9c0f839114|local|mta|auth|grafana@mailhost.cus.int|recipient@example.com|recipient@example.com|1844442211|1844442211|1644441235|0|inflight|0|

/var/log/mail.log¶

Logs for a successfully relayed email. The most important part is the stat="250 Ok at the end, which is the answer of the relaying mailserver (gmail.com in this example) saying: Ok, I have taken this email and will relay it accordingly.

tail -n 0 -f /var/log/mail.log
# Now send the email and watch the logs
smtp connected address=local host=mail.example.com
smtp message msgid=d994c46d size=464 nrcpt=1 proto=ESMTP
smtp envelope evpid=d994c46d2502c97b from=<grafana@mailhost.cus.int> to=<recipient@example.com>
smtp disconnected reason=quit
mta connecting address=smtp+tls://1.2.3.4:587 host=gmail.com
mta connected
mta tls ciphers=TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
mta server-cert-check result="success"
mta delivery evpid=d994c46d2502c97b from=<monitoring@example.com> to=<recipient@example.com> rcpt=<-> source="3.4.5.6" relay="1.2.3.4 (gmail.com)" delay=1s result="Ok" stat="250 Ok 0107017ee057c78c-d9f1e0e9-a73b-48e0-bb5b-430640e0edd7-000000"

Logs for an invalid MAIL FROM. Note that opensmtpd complains about the RCTP TO, even though the MAIL FROM is the problem, as there is no match defined for this MAIL FROM:

smtp connected address=local host=mail.example.com
smtp failed-command command="RCPT TO:<recipient@example.com> " result="550 Invalid recipient: <recipient@example.com>
smtp disconnected reason=disconnect

Logs for an unreachable relay-server. In this case the local firewall blocks the connection (connection refused):

smtp connected address=local host=mail.example.com
smtp message msgid=07ecd737 size=464 nrcpt=1 proto=ESMTP
smtp envelope evpid=07ecd737503da7ae from=<grafana@mailhost.cus.int> to=<recipient@example.com>
smtp disconnected reason=quit
mta connecting address=smtp+tls://1.2.3.4:587 host=gmail.com
mta error reason=IO Error: Connection refused
smtp-out: Disabling route [] <-> 1.2.3.4 (gmail.com) for 15s

Logs for a mailserver relay that has a bad SSL certificate (sadly very common):

smtp connected address=local host=mail.example.com
smtp message msgid=8d2b2bfd size=518 nrcpt=1 proto=ESMTP
smtp envelope evpid=8d2b2bfd60715eac from=<monitoring@example.com> to=<recipient@example.com>
mta connecting address=smtp+tls://1.2.3.4:587 host=gmail.com
smtp disconnected reason=quit
mta connected
mta tls ciphers=TLSv1.2:DHE-RSA-AES256-SHA256:256
mta error reason=SSL certificate check failed
smtp-out: Disabling route [] <-> 1.2.3.4 (gmail.com) for 15s

The fix for this is to set tls no-verify in the action like so:

action "example_bad_tls_relay" \
    relay \
        tls no-verify \
        host smtp+tls://website@gmail.com:587 \
    auth <relay_secrets> \
    mail-from website@example.com

/etc/smtpd.conf in detail¶

This example /etc/smtpd.conf goes into detail about the relevant configurations:

# Read an opensmtpd "table" (a table is like a small database file)
# Doc: https://man.openbsd.org/table
# The contents of the table "relay_secrets" declare a purpose to a specific smtp-user and -password pair
# Example:
# purpose    user:password
# monitoring monitoring@example.com:secretpassword
table relay_secrets file:/etc/smtpd/tables/relay_secrets


# Declare an opensmtpd "filter"
# Doc: https://man.openbsd.org/smtpd.conf#MAIL_FILTERING
# A filter is used to either accept, reject or modify an email
filter regex_replace_data proc-exec "/etc/smtpd/filters/regex-replace-data.py"


# Only open the port on 127.0.0.1 on the loopback interface
# Doc: https://man.openbsd.org/smtpd.conf#listen
listen on lo \
    # Listen on tcp port 25
    # Doc: https://man.openbsd.org/smtpd.conf#port
    port 25 \
    # When accepting a SMTP dialogue, announce yourself as mail.example.com
    # Also write mail.example.com as processing mailserver into the email header
    # This hostname is also set in /etc/mailname
    # Doc: https://man.openbsd.org/smtpd.conf#hostname
    hostname mail.example.com \
    # Apply the filter "regex_replace_data" to all emails
    # Doc: https://man.openbsd.org/smtpd.conf#filter~5
    filter regex_replace_data


# Listen on /var/run/smtpd.sock for SMTP connections
# Doc: https://man.openbsd.org/smtpd.conf#listen~2
listen on socket \
    # Omit the from part when prepending “Received” headers
    # Doc: https://man.openbsd.org/smtpd.conf#mask-src~2
    mask-src \
    # Assign the connection this specific tag (only used inside opensmtpd, not written to the email header)
    # Doc: https://man.openbsd.org/smtpd.conf#tag~2
    tag "socket" \
    # Apply the filter "regex_replace_data" to all emails
    # Doc: https://man.openbsd.org/smtpd.conf#filter~5
    filter regex_replace_data


# Define the action "relay_monitoring"
# Doc: https://man.openbsd.org/smtpd.conf#action
action "relay_monitoring" \
    # This action relays emails to another SMTP server
    # Doc: https://man.openbsd.org/smtpd.conf#relay
    relay \
    # Require TLS when relaying
    # Doc: https://man.openbsd.org/smtpd.conf#tls
    tls \
    # Relay the email to over this particular host
    # Doc: https://man.openbsd.org/smtpd.conf#host
    # Note the <monitoring> string, which refers to the line in the table "relay_secrets" we defined above!
    # monitoring@example.com is NOT an email address! It is the syntax <relay_secrets_table_line>@host:port
    host smtp+tls://monitoring@example.com:587 \
    # For the <monitoring> relay credentials defined in the previous line, use the table relay_secrets to look up the credentials
    # Doc: https://man.openbsd.org/smtpd.conf#auth
    auth <relay_secrets> \
    # Use the given email address as the MAIL FROM address in the SMTP transaction
    # Doc: https://man.openbsd.org/smtpd.conf#MAIL FROM
    MAIL FROM monitoring@example.com


# When tools, webapps, linux users and others try to relay email over opensmtpd, they always state a "MAIL FROM" line in the SMTP transaction.
# "match" lines associate MAIL FROM with a particular relay
# Doc: https://man.openbsd.org/smtpd.conf#match
match \
    # Connection can only come from a local IP (127.0.0.1)
    from local \
    # Match "MAIL FROM: root"
    MAIL FROM "root" \
    # Match any destionation
    for any \
    # Use the action "relay_monitoring" defined above
    action "relay_monitoring"

# More examples
match from local MAIL FROM "prometheus-alertmanager" for any action "relay_monitoring"
match from local MAIL FROM "grafana" for any action "relay_monitoring"

# Reject everything else
match from any reject