TLDR
At work, we introduce a new feature that relies on a new custom header. This custom header acts as a toggle. The new frontend code makes the request to the backend api with this custom header. Curiously the backend didn’t behave as expected even when the toggle is turned on. Upon investigation, the root cause is a CDN misconfiguration where the old OPTIONS response is cached and returned to the frontend, causing the new custom header to always be ommitted from the list of allowed headers.
Now, what does a frontend making a request to backend have to do with OPTIONS?
CORS
The web frontend and API backend are hosted on 2 different subdomains. In this case, most requests will be classified as CORS (Cross-Origin Resource Sharing). Yeah.. the dreaded CORS. Browsers make a “preflight” request to the backend API, in order to check that it will permit the actual request. In that preflight, the browser sends headers that indicate the HTTP method and headers that will be used in the actual request.
Preflight request is an OPTIONS request made your browser before your non-simple request. POST, PUT, DELETE are examples of non-simple requests.
Preflight request carries headers that indicate the HTTP method (GET, PUT, POST, DELETE) and other headers that will be used in the actual request. The server responds whether the actual POST request is safe to send.
Let’s take a look at an example preflight response:
OPTIONS /resource/foo
HTTP/1.1 200 OK
Access-Control-Request-Method: POST
Access-Control-Request-Headers: x-requested-with
Origin: https://foo.bar.org
In this response, the server indicates that only POST requests made from https://foo.bar.org is supported for this path. Additionally, the server only accepts x-requested-with header along with other CORS-safelisted request header (Accept, Accept-Language, Content-Language, Content-Type, Range).
If the original request is a DELETE /resource/foo then the browser will block the request and show a CORS error in the browser console. If the original request contains custom headers other than x-requested-with then these headers are stripped from the request.
Because the OPTIONS response is cached on the CDN layer, our new frontend code will receive the old OPTIONS response which didn’t contain this new custom header. So, the custom header is never passed to the backend API.
That said, there are valid scenarios where you can cache the OPTIONS response, e.g. if the list of custom headers are fixed.
Appendix
It is unsurprising AWS Cloudfront behaviour defaults to not caching OPTIONS and have an explicit toggle to turn on OPTIONS cache.