Setting Dynamic nginx Cache Times based on Custom URI or URL

Sometimes you may want to set dynamic cache times for different URLs on your web site. For example if your blog content rarely changes you may want the cache expiration to be a day but if you have a page that is constantly updated, you may want a microcache where you only cache it for 1- 5 minutes so your server can handle bursts in traffic. If you have ever tried to use a variable in nginx for the fastcgi_cache_valid directive you will quickly find it throws an error! This post shows you how to set dynamic cache times based on different URLs on your server.

If you want to improve the skip cache reasons for nginx see this post

Setting Dynamic nginx Cache Times based on Custom URI or URL

The nginx error message when trying to use a variable for fastcgi_cache_valid looks like this

nginx: [emerg] invalid time value "$cache_ttl" in /etc/nginx/sites-enabled/www.example.com:86

It turns out you cannot use variables for the fastcgi_cache_valid directive!

Luckily nginx lets you set dynamic cache expiration times based on URI or any other value you can make an if statement for with the X-Accel-Expires header.

This X-Accel-Expires header lets you tell nginx how long to store the fastcgi or proxy cache for on the file or storage system.

The first step is to set a variable for the cache’s TTL (time to live) which will be our default cache expiration time in seconds. Here it is 3600 seconds which is 1 hour.

#cache should be enabled by default
set $skip_cache 0;
#set default cache TTL to 1 hour in seconds for X-Accel-Expires header
set $cache_ttl 3600;

Then we need to add a snippet for changing the $cache_ttl, here I am setting a microcache time of 120 seconds for the WordPress REST API which always contains the string wp-json. To be explicit I am also adding the $skip_cache variable.

# Set the REST API ttl to 2 minutes so it acts as a microcache
if ($request_uri ~* "wp-json") {
	set $cache_ttl 120;
	set $skip_cache 0;
}

For debugging purposes it is a good idea to add the $cache_ttl value as a header as well so we can see if it is working correctly.

Here I called it the WP-Bullet-Cache-TTL header

We also need to define the X-Accel-Expires header as $cache_ttl which is done with the set $http_x_accel_expires $cache_ttl; line

location ~ \.php$ {
    try_files $uri =404;
    # add cache status
    add_header WP-Bullet-Fastcgi-Cache $upstream_cache_status;
    # add the cache skip reason if relevant
    add_header WP-Bullet-Skip $skip_reason;
    # add cache expiration header
    add_header WP-Bullet-Cache-TTL $cache_ttl;
    include fastcgi_params;
    fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    fastcgi_split_path_info ^(.+\.php)(.*)$;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_cache_bypass $http_secret_header $skip_cache;
    fastcgi_no_cache $skip_cache;
    fastcgi_cache WORDPRESS;
    fastcgi_cache_valid 404      1m;
    fastcgi_cache_valid  60m;
    # set the X-Accel-Expires header to the cache_ttl variable
    set $http_x_accel_expires $cache_ttl;
}

Here is what the whole nginx virtual host could look like for fastcgi caching with dynamic cache TTL expiration times for specific URL strings.

#cache should be enabled by default
set $skip_cache 0;
#set default cache TTL to 1 hour in seconds for X-Accel-Expires header
set $cache_ttl 3600;

# POST requests and urls with a query string should always go to PHP
if ($request_method = POST) {
	set $skip_cache 1;
	set $skip_reason POST;
}

if ($query_string != "") {
	set $skip_cache 1;
	set $skip_reason QueryString;
}

# Don't cache uris containing the following segments
if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php|/feed/|index.php|sitemap(_index)?.xml") {
	set $skip_cache 1;
	set $skip_reason URI;
}

# Don't use the cache for logged in users or recent commenters
if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_no_cache|wordpress_logged_in") {
	set $skip_cache 1;
	set $skip_reason Cookie;
}

# Set the REST API ttl to 2 minutes so it acts as a microcache
if ($request_uri ~* "wp-json") {
	set $cache_ttl 120;
}

location / {
	try_files $uri $uri/ /index.php?$args;
}

# Deny public access to wp-config.php
location ~* wp-config.php {
	deny all;
}

location ~ \.php$ {
	try_files $uri =404;
	# add cache status
	add_header WP-Bullet-Fastcgi-Cache $upstream_cache_status;
	# add the cache skip reason if relevant
	add_header WP-Bullet-Skip $skip_reason;
	# add cache expiration header
	add_header WP-Bullet-Cache-TTL $cache_ttl;
	include fastcgi_params;
	fastcgi_pass unix:/run/php/php7.0-fpm.sock;
	fastcgi_split_path_info ^(.+\.php)(.*)$;
	fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
	fastcgi_cache_bypass $http_secret_header $skip_cache;
	fastcgi_no_cache $skip_cache;
	fastcgi_cache WORDPRESS;
	fastcgi_cache_valid 404 1m;
	fastcgi_cache_valid 60m;
	# set the X-Accel-Expires header to the cache_ttl variable
	set $http_x_accel_expires $cache_ttl;
}

Testing nginx Dynamic Cache Times

Using cURL

curl -I https://wp-bullet.com/wp-json/

Output, notice the bottom headers

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 09 Sep 2018 13:09:57 GMT
Content-Type: application/json; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Robots-Tag: noindex
Link: <https://wp-bullet.com/wp-json/>; rel="https://api.w.org/"
X-Content-Type-Options: nosniff
WP-Bullet-Fastcgi-Cache: MISS
WP-Bullet-Cache-TTL: 120

Now let’s test the homepage

curl -I https://wp-bullet.com/

Output

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 09 Sep 2018 13:13:00 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Link: <https://wp-bullet.com/wp-json/>; rel="https://api.w.org/"
WP-Bullet-Fastcgi-Cache: HIT
WP-Bullet-Cache-TTL: 3600

Tadda!

Controlling multiple Cache Times with nginx using map

Thank you David for pointing out that map can also be used for fine grained control over the nginx cache expiration. This would be especially useful if you have a lot of exceptions!

map $request_uri $cache_ttl {
    default "3600";
    "~*wp-json" "120";
    # set other variables
}

With this method you could potentially drop the if statement as long as the flow of the caching logic still makes sense.

Sources

nginx X-Accel Wiki
Change fastcgi_cache valid for response codes
nginx proxy_ignore_headers Documentation