{"id":2173,"date":"2016-03-06T12:47:45","date_gmt":"2016-03-06T18:47:45","guid":{"rendered":"http:\/\/www.mooreds.com\/wordpress\/?p=2173"},"modified":"2021-11-21T12:39:41","modified_gmt":"2021-11-21T18:39:41","slug":"step-back-and-review-the-problem-ssl-edition","status":"publish","type":"post","link":"https:\/\/www.mooreds.com\/wordpress\/archives\/2173","title":{"rendered":"Step back and review the problem: SSL Edition"},"content":{"rendered":"

I spent the better half of a week recently getting an understanding of how SSL works and how to get client certificates, self signed certificates and java all playing nicely. \u00a0However, in the end, taking a big step back and examining the problem led to an entirely different solution.<\/p>\n

The problem in question is access to a secure system from a heroku application. \u00a0The secure system is protected by an IP based firewall, basic authentication over SSL, and client certificates that were signed by a non standard certificate authority (the owner of the secure system).<\/p>\n

We are using HttpClient from Apache for all HTTP communication, and there’s a nice example<\/a> (but, depending on your version of the client, run loadKeyMaterial<\/code> too, as that is what ends up sending the client cert). \u00a0When I ran it from my local machine, after setting the IP to be one of the static IPs, I ended up seeing a wide variety of errors. \u00a0Oh, there are many possible errors! \u00a0But finally, after making sure that the I could import the private key into the store<\/a>, was using the correct cipher\/java version combo<\/a>, writing a stripped down client that didn’t interface with any other parts of the system, learning about the javax.net.debug=ssl<\/code> switch<\/a>, making sure the public certificates were all in the store, and that I had IP based access to the secure system, I was able to watch the system go through a number of the steps outlined here:<\/p>\n

\"SSL
Image from the JSSE Guide<\/a><\/figcaption><\/figure>\n

But I kept seeing this exception: error: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure<\/code>. I couldn’t figure out how to get past that. From searching there were a number of issues that all manifested with this error message, and “guess and check” wasn’t working. Interestingly, I saw this error only when running on my local machine with the allowed static IP. When I ran on a different computer that was going through a proxy which provided a static IP, the client certificate wasn’t being presented at all.<\/p>\n

After consulting with the other organization and talking with members of the team about this roadblock, I took a step back and validated the basics. I read the JSSE Reference guide<\/a>. I used the openssl tools to verify that the chain of certificates was valid<\/a>. (One trick here is that if you have intermediate certs in between your CA and the final cert, you can add only one -untrusted<\/code> switch. So if you have multiple certs in between, you need to combine the PEM files into one file.) I validated that the final certificate in the keystore matched the CSR and the private key<\/a>. Turns out I had the wrong key, and that was the source of the handshake issue. Doh!<\/p>\n

After I had done that, I took a look at the larger problem. Heroku doesn’t guarantee IP addresses, not even a range<\/a>. There are add on solutions (proximo, quotaguard, fixie) that do provide a static IP address, and that was our initial plan. However, all of these are proxy based solutions. A quick search turns up the unpleasant reality that proxies can’t pass client certificates<\/a>. The post talks about a reverse proxy, but it applies as well to regular proxies. From the post:<\/p>\n

Yes, because a client can only send its certificate by using encrypted and
\nSIGNED connection, and only the client can sign the certifikate (sic) so server
\ncan trust it. The proxy does not know the clients private key, otherwise the
\nconnection would not be secure (or not in the way most people know that).
\n…
\nSSL is made up to avoid man-in-the-middle attack, and the reverse proxy IS
\nthe man-in-the-middls. Either you trust it (and accept what it sends) or
\ndon’t use it.<\/p><\/blockquote>\n

All my work on having the java code create the client certificate was a waste. Well, not a total waste because now I understood the problem space in a way I hadn’t before, so I could perform far better searches.<\/p>\n

I opened a support request with our proxy provider, but it was pretty clear from the internet, the support staff and the docs that this was a niche case. I don’t know if any static IP proxy providers support client certificates, but I wasn’t able to find one.<\/p>\n

Instead, we were able to use AWS elastic IP and nginx to set up our own proxy<\/a>. Since we controlled it, we could install the client certificate and key on it, and have the heroku instance connect to that proxy. (Tips for those instructions–make sure you download the openssl source, as nginx wants to compile it into the web server. And use at least version 1.9 of the community software.)<\/p>\n

So, I made some mistakes in this process, and in a personal retro<\/a>, I thought it’d be worth talking about them.<\/p>\n