tl;dr: Yes, it did. It was a configuration issue.
I deployed my app using Kamal, a tool from Basecamp that makes deployment to multiple VMs easy. I also run a few different apps under the same domain. So, I set up a nginx reverse proxy to sit between users and my VM and pointed the domain to that nginx, something like this:
sequenceDiagram
participant User
participant Nginx
participant App
User ->> Nginx: HTTPS Request
Nginx ->> App: HTTP Request
App -->> Nginx: HTTP Response
Nginx -->> User: HTTPS Response
So far so good.
One day, I wanted to get the user’s IP address from my app, so I set up the following nginx config
location ^~ /subdomain/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Referer $http_referer;
proxy_set_header Host $http_host;
proxy_pass http://10.100.0.5;
}
Using the proxy_set_header
directive, nginx adds extra headers to the original request before passing it to my app. In my app, I was expecting to get the user’s IP in my X-Real-IP
headers, but instead, I got the IP of the nginx server. Huh?
I tried showing the nginx variables to see what values $remote_addr
point to in production. I got my client IP as expected.
I tried setting a different header, e.g. Test-Header
, to see if it would work. Interestingly, it works. My app could read that custom header.
Did the order of proxy_set_header
matter? Most examples I found on the internet have the proxy_pass
as the first line. Nope, the order didn’t matter. I also tried setting proxy_pass_request_headers on
to no avail. I still can’t get my app to read the client ip from X-Real-IP
.
At this point, I’m ready to give up and just set a custom header for my app to read. But there’s a nagging feeling: What is wrong? Did Kamal eat my X-Forwarded-* headers?
Well, at this point, I remember to mind the layers.
The architecture I show you at the beginning of this article is right, but we are missing something.
Enter Traefik. Kamal uses Traefik to load balance requests between multiple VMs running your apps. Even though I only have a single VM to run my app, Kamal still provisions a Traefik instance. By default, Traefik doesn’t trust X-Forwarded-*
headers. It will override its value. Bam, now we know why our app could only receive nginx’s IP address. From Traefik’s point of view, nginx is the “user”.
sequenceDiagram
participant User
participant Nginx
participant Traefik
participant App
User ->> Nginx: HTTPS Request
Nginx ->> Traefik: HTTP Request
Traefik ->> App: HTTP Request
App -->> Traefik: HTTP Response
Traefik -->> Nginx: HTTP Response
Nginx -->> User: HTTPS Response
So, how do we fix this?
Add the following lines to your kamal config:
traefik:
args:
entryPoints.web.address: ":80"
entryPoints.web.forwardedHeaders.trustedIPs: "10.100.0.0/24"
and run kamal traefik reboot
to restart your traefik container with the new args. You can read about the trustedIPs configuration here
That’s it. We have solved the mystery.
P.S: Please, don’t talk about the time I lost on this little issue :)