Monitoring Postfix Email Queue with Push Notifications

 

Postfix Mail Queue graph in munin

Sending and receiving emails is by default done asynchronously. Both are handled by your email service like postfix, Exim, Sendmail, etc. We will focus on postfix, but you can map this on any email system by simply replacing the line for accessing the mail queue.

As the mail service process them asynchronously, they need to be queued before and maybe during processing for good reasons:

By using queues, you should have some crucial things in mind like it is explained in Wikipedia: Queueing theory:

Let’s focus on the queue size in our email service and how to monitor this number. But first, we need to know how to access the postfix email queues and what we can do with them.

Accessing Queues and Email Messages in Queues

First of all, we want to list enqueued emails in our system. For postfix, we can do this with one of these commands:

root@server $ mailq
Mail queue is empty
root@server $ postqueue -p
Mail queue is empty

“Mail queue is empty” means there is nothing to process. To see some enqueued mail, you can try this from your command line:

root@server $ mail -s 'testing message subject' someone@gsmail.com <<< 'testing message body' && mailq
-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
2A0324780066      251 Wed Nov  4 13:29:07  root
                                         someone@gsmail.com

-- 0 Kbytes in 1 Request.

Now you can see an enqueued email. If your queue has some other mails, you will see them too. If you choose a not existing domain as I did with “gsmail.com” you can rerun the command to see processing information for your email:

root@server $ postqueue -p
-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
E900C4780073      462 Wed Nov  4 13:36:46  root@mail.myserver.local
(Host or domain name not found. Name service error for name=gsmail.com type=MX: Host not found, try again)
                                         someone@gsmail.com

As we have some unprocessable email in a queue, we can try to access the message by Queue ID “E900C4780073” and see what is inside:

root@server $ postcat -q E900C4780073
*** ENVELOPE RECORDS deferred/E/E900C4780073 ***
message_size:             462             210               1               0             462               0
message_arrival_time: Wed Nov  4 14:36:46 2020
create_time: Wed Nov  4 14:36:46 2020
named_attribute: rewrite_context=local
sender_fullname: root
sender: root@mail.myserver.local
*** MESSAGE CONTENTS deferred/E/E900C4780073 ***
Received: by mail.myserver.local (Postfix, from userid 0)
	id E900C4780073; Wed,  4 Nov 2020 14:36:46 +0100 (CET)
To: someone@gsmail.com
Subject: testing message subject
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Message-Id: <20201104133646.E900C4780073@mail.myserver.local>
Date: Wed,  4 Nov 2020 14:36:46 +0100 (CET)
From: root@mail.myserver.local (root)

testing message body
*** HEADER EXTRACTED deferred/E/E900C4780073 ***
named_attribute: encoding=8bit
named_attribute: dsn_orig_rcpt=rfc822;someone@gsmail.com
original_recipient: someone@gsmail.com
recipient: someone@gsmail.com
*** MESSAGE FILE END deferred/E/E900C4780073 ***

Deleting emails from postfix queue

To remove unwanted or for testing generated emails from the postfix queue, you can use the command postsuper and the Queue ID to remove a message:

root@server $ postsuper -d E900C4780073
postsuper: E900C4780073: removed
postsuper: Deleted: 1 message

Be careful when viewing and deleting emails, as you could be breaking the law. But we need all these commands to test our monitoring system for postfix queues. Keep on reading.

Script to monitor the queue size

First of all, we want to write a bash script to see the queue size. We use absolute paths to commands to reduce possible errors due to missing paths in $PATH. Please consider adjusting the paths for your system. By executing these commands, we can see the queue size:

root@server $ /usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}'

root@server $

If our queue is empty, we get an empty result. Now let’s enqeue an email like we did above and see what the script will tell us:

root@server $ mail -s 'testing message subject' someone@gsmail.com <<< 'testing message body'
root@server $ /usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}'
1

We enqueued an email, and now we can see “1” as a return value - great! Now we write a simple bash script to print out a message if the queue is bigger than 0:

#!/bin/bash

mailq_count="$(/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}')"

# If variable is empty, then the queue is empty -> set it to zero
if [ -z "$mailq_count" ]; then
  mailq_count=0
fi

if [ "$mailq_count" -gt 0 ]; then
  echo "Mail count on mail.myserver.local is ${mailq_count}" | /usr/bin/mail -s "mail.myserver.local queue size: ${mailq_count}" notify.me@somemail.local
fi

This script is pretty simple and should send an email if the mail queue is greater than 0. You can adjust the number for your needs. For low traffic servers, something between 10 and 20 should be the right choice.

Now you can adjust the script for your needs and use the commands above to put some messages in a queue and remove them for testing.

You can replace the notification mail line for simpler testing:

if [ "$mailq_count" -gt 0 ]; then
  echo "Mail count on mail.myserver.local is ${mailq_count}"
fi

Try it for yourself and meet your expectations.

Running the script by a cron job

Since our script is working, we want to call it every minute and notify us if the queue exceeds our preset value. We can do this through a cron job entry in a new file like /etc/cron.d/mailqueue-watchdog:

*/1 *  * * *   root    nice -n 19  /usr/local/sbin/mailqueue_size.sh

You can use the commands above and add or delete some emails for testing with a not existing domain, as I did with domain name gsmail.com.

Getting push notifications

Receiving emails to be notified if the queue is in trouble is not the best decision as an admin gets many emails; that’s why we want to use push notifications through a different service.

You can use Pushover or any other service for sending a copy of the queue size email as a push message to your favorite smartphone. We don’t want to trust another service in such a critical situation fully; that’s why we’ve set up to receive an email and a push notification too.

Before Pushover I used IFTTT to receive push notifications, which should be fine too.

Happy monitoring and hopefully always low queue sizes!

Tags