Mutual TLS Authentication

Mutual authentication or two-way authentication refers to two parties authenticating each other at the same time. By default the TLS protocol only proves the identity of the server to the client using X.509 certificate and the authentication of the client to the server is left to the application layer. TLS also offers client-to-server authentication using client-side X.509 authentication.

When Mutual TLS authentication is in place, both client and server authenticate each other through the digital certificate so that both parties are assured of the others' identity. In this aspect, both client and server use 12 handshake messages to establish the encrypted channel prior to message exchanging.

Refer to https://www.codeproject.com/articles/326574/an-introduction-to-mutual-ssl-authentication

When set up Mutual TLS authentication, it involves creating your own Certification Authority, self-signing the server certificate and client certificate, then installs self-signed certificate on server and client. The detailed steps are as followings:

1. Generate CA certificate:
1.1. Generate CA key:
openssl genrsa -out ca.key 2048
1.2. Generate CA certificate request:
openssl req -new -key ca.key -out ca.csr -subj /C=US/ST=ABC/L=XYZ/O=Example/OU=Test/CN=Root/emailAddress=john.doe@example.com
1.3. Generate CA certificate:
openssl x509 -req -days 3650 -in ca.csr -signkey ca.key -out ca.pem

2. Generate server certificate:
2.1. Generate server key:
openssl genrsa -out server.key 2048
2.2. Generate certificate request:
openssl req -new -key server.key -out server.csr -subj /C=US/ST=ABC/L=XYZ/O=Example/OU=Test/CN=test.cert.example.com/emailAddress=john.doe@exampl
e.com
2.3. Generate certificate:
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out test.cert.example.com.pem -days
365

3. Generate client certificate:
3.1. Generate client key:
openssl genrsa -out client.key 2048
3.2. Generate certificate request:
openssl req -new -key client.key -out client.csr -subj /C=CN/ST=FJ/L=City/O="Company"/OU=Dept/CN=ClientID/emailAddress=support@company.com
3.3. Generate certificate:
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out client.pem -days 365

4. Combine cert and key:
cat client.pem client.key > client2.pem

Some client like Firefox, to install the client certificate, we’ll need a PKCS#12 file which stores both the certificate and the client’s private key.
openssl pkcs12 -export -clcerts -in client.pem -inkey client.key -out client2.p12

5. Configure server:
Apache:
TLSCACertificateFile ca.pem
TLSCertificateKeyFile server.key
TLSCertificateFile test.cert.example.com.pem
TLSVerifyClient demand

Nginx:
ssl_certificate test.cert.example.com.pem;
ssl_certificate_key server.key;
ssl_client_certificate ca.pem;
ssl_verify_client on;

6. Configure client (device or browser):
6.1 Add ca.pem as a trusted CA certificate on client side
6.2 Add client2.pem or client2.p12 as a client certificate and use it

https://blog.codeship.com/how-to-set-up-mutual-tls-authentication/ also gives step-by-step instructions to set up mutual authentication using openssl commands.

CA certificate
openssl genrsa -aes256 -out ca/ca.key 4096 chmod 400 ca/ca.key
openssl req -new -x509 -sha256 -days 730 -key ca/ca.key -out ca/ca.crt
chmod 444 ca/ca.crt
openssl x509 -noout -text -in ca/ca.crt

Server certificate
openssl genrsa -out server/client-ssl.bauland42.com.key 2048
chmod 400 server/client-ssl.bauland42.com.key
openssl req -new -key server/client-ssl.bauland42.com.key -sha256 -out server/client-ssl.bauland42.com.csr
openssl x509 -req -days 365 -sha256 -in server/client-ssl.bauland42.com.csr -CA ca/ca.crt -CAkey ca/ca.key -set_serial 1 -out server/client-ssl.bauland42.com.crt
chmod 444 server/client-ssl.bauland42.com.crt
openssl x509 -noout -text -in server/client-ssl.bauland42.com.crt
openssl verify -CAfile ca/ca.crt server/client-ssl.bauland42.com.crt

Client certificate
openssl genrsa -out client/heiko.key 2048
openssl req -new -key client/heiko.key -out client/heiko.csr
openssl x509 -req -days 365 -sha256 -in client/heiko.csr -CA ca/ca.crt -CAkey ca/ca.key -set_serial 2 -out client/heiko.crt
openssl pkcs12 -export -clcerts -in client/heiko.crt -inkey client/heiko.key -out client/heiko.p12

Nginx server config
ssl_certificate /etc/nginx/certs/server/client-ssl.bauland42.com.crt;
ssl_certificate_key /etc/nginx/certs/server/client-ssl.bauland42.com.key;
ssl_client_certificate /etc/nginx/certs/ca/ca.crt;
ssl_verify_client on;