We had some intermittent problems recently with a mail server not being available. As it turns out, I basically was doing a “denial of service” attack on my own mail server, thanks to some caveats in how IMAP, push notifications, and my mobile email client, K9 work together.
The problem
One of the mail servers we operate quite regularly became unavailable, apparently not accepting any connections. We managed to investigate the problem as it was happening, and found out all available connections for the IMAP server were in use, and nearly all of them originated from a familiar IP address: my home.
I disconnected the email clients on my devices (laptop, phone, tablet) and switched them on one by one. Clearly, my Android-based phone and tablet with the K9 mail client were causing the problem: they made dozens of connections. It turned out the problem was even worse: K9 reconnects several times.
IMAP and push notifications
Normally, an email client connects to the mail server to check if there is new mail in any of the folders, via IMAP. It would be nicer if the server just informs the client if there is a new mail available. The client will know immediately, and doesn’t have to poll the server so often, which reduces the load on the server too.
Such “push notifications” are not part of the original IMAP protocol. There are two ways in which IMAP tries to solve this:
- Keep the connection between client and server open with IDLE commands, so that the server can talk to the client.
- Extend the IMAP protocol with “IMAP NOTIFY“, that both server and client need to implement, to offer more sophisticated push notifications over a single connection.
Sadly, the second option is still under development, and clients and servers (such as Cyrus) haven’t really implemented it yet.
And even more sadly: with IDLE, you need an open connection for each folder that needs push notifications. If you only have push notifications enabled on one Inbox folder, that’s not such a big deal, but to have push notifications on a dozen folders or more, you need a dozen or more open connections between server and client.
How K9 and Thunderbird deal with it
Apparently, Thunderbird limits itself by default to five open connections to the server, no matter how many folders you have. As far as I found out, you’ll have push notifications on the last five folders you’ve looked at.
But K9 does not limit itself. In my case, with a few accounts on our server, and with “push” enabled on all my preferred (“First class”) folders, that resulted in a few dozen open connections.
To make things even worse, K9 quickly detects when I switch networks, for instance when going from Wifi to 3G, and establishes new connections for those folders over the new network. While the mail server still has the old connections open, waiting for them to time out before closing them.
How I “DDOS” my own mail server
On a regular day, I start at home, then go to my office, with all my devices.
- Phone and tablet: 2 devices
- Running K9, each with perhaps 20 connections for various folders
- Switching from my home Wifi to 3G, then to my office Wifi, usually within 10-15 minutes: 3 networks
That adds up pretty quickly: 2 * 20 * 3 = 120 connections. No wonder my colleague had trouble accessing email as soon as I started working.
Suffice to say: I switched off push notifications.
Thank you so much for this post. I have been trying to figure out what’s wrong with my Cyrus IMAPD for more than a month. Having disabled push in K9 on all my mobile devices, things are back to normal.
The worst thing is that the Cyrus logs are completely useless in this case. They only show TLS negotiations failing (mostly due to timeouts, but that’s for you to figure out) which had me checking all the usual crypto stuff to no avail.
Another pitfall: My Cyrus handles one K9 with push notifications just fine. So when I started using K9 on more and more devices, Cyrus began to fail without me noticing that this had to with the additional K9 installations.
Glad it was of help, thanks for the feedback!