Haproxy, Cloudfront, and S3
Problem:
I was attempting to setup an haproxy reverse proxy to an s3 static site fronted by Cloudfront, something like this
Haproxy reverse proxy => Cloudfront distribution w/ redirect HTTP to HTTPS => S3 Static Site
However with this backend config
backend broken_backend mode http http-request set-header Host my_cloudfront_domain.com server broken_backend my_cloudfront_domain.com
I was getting cryptic ~400 unable to handle request~ errors.
Solution:
The solution ended up being all in my haproxy config. After a some google searching I learned that using HTTPS backends in haproxy required some extra config (and haproxy 1.5 or newer). One solution is to not check the validity of a servers certificate. You should only do if you trust the server, in my case I do. The config looks something like this:
backend still_broken_backend mode http http-request set-header Host my_cloudfront_domain.com server still_broken_backend my_cloudfront_domain.com:443 ssl verify none
However then I started to get 503's, which turned out to be a little more nuanced. The solution was to send a Server Name Identification (SNI). Without it Cloudfront doesn't which name you're trying to connect to, and as a result which SSL certificate to use. Cloudfront handles this problem just close the connection, causing haproxy to throw a 503.
Everything together looks like this:
backend working_backend mode http http-request set-header Host my_cloudfront_domain.com server working_backend my_cloudfront_domain.com:443 ssl verify none sni str(my_cloudfront_domain.com)
I wish this section in the docs would have shown up earlier in my research, the phrase "SNI" ended up being the key to unraveling all this.. Maybe this article will help google's magic algorithm connect the dots a little better.