How to detect and fix (mitigate) SSL/TLS renegotiation DOS vulnerability in Postfix

Vulnerability scanners, such as OpenVAS, might report a "SSL/TLS renegotiation DoS vulnerability" on the SMTP protocol.

But what kind of vulnerability is this, how can it be detected and how can it be solved (mitigated) in Postfix? That's why this article was created.

SSL/TLS Renegotiation: Potential DoS

Most of the security vulnerabilities (these days) point to weaknesses in applications. This could be cross site scripting (XSS), code injections (e.g. SQL injections), authentication bypass and many more. But SSL/TLS Renegotiation is another kind of vulnerability, a Denial of Service (DoS) vulnerability.

A TLS renegotiation is basically re-establishing a connection to a server with SSL/TLS encryption. Renegotiation allows the current connection to re-establish the connection - over and over again. Doesn't sound too bad, does it? The problem is however, that the server side needs to use more CPU to renegotiate the connection (much more than the client). The excellent TLS computational DoS mitigation document written by Vincent Bernat, explains the vulnerability in big depth and also mentions:

a server is using 100 times more processing power than the client

This can be used to run SSL/TLS renegotiation attacks, leading to higher CPU usage on the server, potentially slowing down the application(s) on the server - until the application(s) time out and stop responding altogether.

How to detect the SSL/TLS Renegotiation vulnerability

The vulnerability can be detected and verified using the openssl s_client sub-command. Once the connection to a server is established, the " R " input initiates a Renegotiation of the connection. If the server responds, the connection was Renegotiated - meaning the vulnerability exists. If the server cuts the connection after the " R " input, the vulnerability does not exist.

Web servers, such as Apache or Nginx, have disabled the SSL/TLS renegotiation. Initiating a renegotiation results in an error and the connection is cut:

ck@mintp ~ $ openssl s_client -connect webmail.example.com:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = webmail.example.com
verify return:1
---
[. ]
---
read R BLOCK
R
RENEGOTIATING
140113851618624:error:1420410A:SSL routines:SSL_renegotiate:wrong ssl version. /ssl/ssl_lib.c:2133:

In Postfix, the renegotiation is possible and can be verified:

ck@mintp ~ $ openssl s_client -connect mail.example.com:25 -starttls smtp
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mail.example.com
verify return:1
---
[. ]
---
250 SMTPUTF8
R
RENEGOTIATING
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mail.example.com
verify return:1
R
RENEGOTIATING
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mail.example.com
verify return:1
quit
221 2.0.0 Bye
closed

As you can see above, the R input can be used multiple times, the connection is always renegotiated. Hence the potential DoS vulnerability does exist on that mail server.

How to fix (mitigate) SSL/TLS Renegotiation in Postfix

But Postfix wouldn't be world's best SMTP server if it could not handle this or offer a configuration mitigation for this vulnerability.

Postfix' tls_ssl_options configuration option is by default empty, but allows multiple values. One of these values: NO_RENEGOTIATION.

NO_RENEGOTIATION
Postfix >= 3.4. This can reduce opportunities for a potential CPU exhaustion attack. See SSL_CTX_set_options(3).

Apply this to the Postfix' main.cf:

root@mailserver:~# grep tls_ssl_options /etc/postfix/main.cf
tls_ssl_options = NO_RENEGOTIATION

And reload Postfix:

root@mailserver:~# service postfix reload

Is the vulnerability still active? Let's test:

The connection is now closed with an error, as soon as a connection renegotiation was attempted.