Did Kamal Eat my X-Forwarded-* Headers?

tl;dr: Yes, it did. It was a configuration issue.

I deployed my app using Kamal, a tool from Basecamp that makes deployment to multiple VMs easy. I also run a few different apps under the same domain. So, I set up a nginx reverse proxy to sit between users and my VM and pointed the domain to that nginx, something like this:

sequenceDiagram
    participant User
    participant Nginx
    participant App
    User ->> Nginx: HTTPS Request
    Nginx ->> App: HTTP Request
    App -->> Nginx: HTTP Response
    Nginx -->> User: HTTPS Response

So far so good.

One day, I wanted to get the user’s IP address from my app, so I set up the following nginx config

location ^~ /subdomain/ {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Referer $http_referer;
    proxy_set_header Host $http_host;
    proxy_pass http://10.100.0.5;
}

Using the proxy_set_header directive, nginx adds extra headers to the original request before passing it to my app. In my app, I was expecting to get the user’s IP in my X-Real-IP headers, but instead, I got the IP of the nginx server. Huh?

I tried showing the nginx variables to see what values $remote_addr point to in production. I got my client IP as expected.

I tried setting a different header, e.g. Test-Header, to see if it would work. Interestingly, it works. My app could read that custom header.

Did the order of proxy_set_header matter? Most examples I found on the internet have the proxy_pass as the first line. Nope, the order didn’t matter. I also tried setting proxy_pass_request_headers on to no avail. I still can’t get my app to read the client ip from X-Real-IP.

At this point, I’m ready to give up and just set a custom header for my app to read. But there’s a nagging feeling: What is wrong? Did Kamal eat my X-Forwarded-* headers?

Well, at this point, I remember to mind the layers.

The architecture I show you at the beginning of this article is right, but we are missing something.

Enter Traefik. Kamal uses Traefik to load balance requests between multiple VMs running your apps. Even though I only have a single VM to run my app, Kamal still provisions a Traefik instance. By default, Traefik doesn’t trust X-Forwarded-* headers. It will override its value. Bam, now we know why our app could only receive nginx’s IP address. From Traefik’s point of view, nginx is the “user”.

sequenceDiagram
    participant User
    participant Nginx
    participant Traefik
    participant App
    User ->> Nginx: HTTPS Request
    Nginx ->> Traefik: HTTP Request
    Traefik ->> App: HTTP Request
    App -->> Traefik: HTTP Response
    Traefik -->> Nginx: HTTP Response
    Nginx -->> User: HTTPS Response

So, how do we fix this?

Add the following lines to your kamal config:

traefik:
  args:
    entryPoints.web.address: ":80"
    entryPoints.web.forwardedHeaders.trustedIPs: "10.100.0.0/24"

and run kamal traefik reboot to restart your traefik container with the new args. You can read about the trustedIPs configuration here

That’s it. We have solved the mystery.

P.S: Please, don’t talk about the time I lost on this little issue :)