How to Cache AJAX GET Requests with Varnish 4

by

Varnish vcl (Varnish Control Language) allows you to be precise about what and how you cache specific URLs in WordPress and WooCommerce. One area which can be a pain for performance enthusiasts is showing up-to-date recent posts in a sidebar widget or show off your popular posts in the footer.

Usually AJAX is used to bypass cache by making a call for PHP to pull fresh data out of the MySQL database. While very convenient, these AJAX requests can be expensive in terms of resources. Luckily these AJAX requests can be cached with Varnish.

The Varnish 3 version of this guide can be found here.

Speed Benefits of Caching AJAX Get Requests with Varnish

When I AJAXify the Recently and WordPress Popular Posts widgets it takes between 200-300ms for the site to display the data. That is not the plugins’ fault, it is just the nature of AJAX (technical explanation here). Luckily we can cache this AJAX request and get it down to 65ms with Varnish 4. I will assume you already have Varnish 4 installed and configured on your VPS or dedicated server.

Here are some pingdom results before and after caching the AJAX requests.

Uncached AJAX Response Time

Using the Recently widget without Varnish took 240 ms.

varnish ajax uncached speed

Varnish cached AJAX Response Time

Using the Recently widget with Varnish took 65 ms.

varnish ajax cached speed

How to Cache AJAX GET Requests with Varnish 4

First, here is a quick explanation of how AJAX requests work.

How AJAX GET Requests Work in WordPress

The diagram Regular AJAX GET Request with WordPress shows how many steps are involved in a typical AJAX request in WordPress.

regular-ajax-get-request-wordpress

The diagram Varnish cached AJAX GET Request with WordPress shows how much faster it is to serve AJAX GET requests with Varnish since MySQL is no longer used.

varnish-cache-ajax-get-request-wordpress

Cache AJAX GET Requests using Varnish 4

Open up your Varnish 4 vcl file

sudo nano /etc/varnish/default.vcl

Add the red sections below, note that these are carefully placed in the vcl to not conflict with the vcl order.

First we need to catch the AJAX requests for non-logged in users before wp-admin requests are passed in sub vcl_recv.

Then we set the time to live in sub vcl_fetch for our captured and cached AJAX GET requests. The snippets are explained in the next section.

...

sub vcl_recv {

...

  # PIPE ALL NON-STANDARD REQUESTS
  # ##########################################################
  if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
      return (pipe);
  }

  # ONLY CACHE GET AND HEAD REQUESTS
  # ##########################################################
  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

##Possibility to cache admin-ajax GET requests
  
  if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
      return (hash);
  }

  # OPTIONAL: DO NOT CACHE LOGGED IN USERS (THIS OCCURS IN FETCH TOO, EITHER
  # COMMENT OR UNCOMMENT BOTH
  # ##########################################################
  if ( req.http.cookie ~ "wordpress_logged_in" ) {
    return( pass );
  }


  # IF THE REQUEST IS NOT FOR A PREVIEW, WP-ADMIN OR WP-LOGIN
  # THEN UNSET THE COOKIES
  # ##########################################################
  if (!(req.url ~ "wp-(login|admin)") 
    && !(req.url ~ "&preview=true" ) 
  ){
    unset req.http.cookie;
  }


  # IF BASIC AUTH IS ON THEN DO NOT CACHE
  # ##########################################################
  if (req.http.Authorization || req.http.Cookie) {
    return (pass);
  }
  
  # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED
  # ##########################################################
  return (hash);

}

# FETCH FUNCTION
# ##########################################################
sub vcl_backend_response {
  # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC 
  # TENDANCY TO SET VARY USER-AGENT.  YOU MAY OR MAY NOT WANT
  # TO DO THIS
  # ##########################################################
  set beresp.http.Vary = "Accept-Encoding";

  # IF NOT WP-ADMIN THEN UNSET COOKIES AND SET THE AMOUNT OF 
  # TIME THIS PAGE WILL STAY CACHED (TTL)
  # ##########################################################

#set the length of time to cache ajax GET requests

if ((bereq.url ~ "admin-ajax.php") && !bereq.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

##set how long to cache for anything that is not admin or ajax

  if (!(req.url ~ "wp-(login|admin)|admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in") {
    unset beresp.http.set-cookie;
    set beresp.ttl = 52w;
    set beresp.grace = 1w;
  }

  if (beresp.ttl <= 0s ||
    beresp.http.Set-Cookie ||
    beresp.http.Vary == "*") {
      set beresp.ttl = 120 s;
      return (hit_for_pass);
  }

  return (deliver);
}

.....

Save the configuration with Ctrl+X, Y and Enter, don’t restart Varnish yet.

Caching AJAX Requests with Varnish Explained

In this Varnish snippet from sub vcl_recv section we are catching AJAX requests for non-logged in users and telling Varnish to look them up in its cache. It is important this snippet comes before you pass wp-admin in Varnish!

##Possibility to cache admin-ajax GET requests
  
 if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    return (hash);
  }

If you want to be even more specific by finding the exact AJAX URL in Chrome’s Developer tools or Firefox Firebug. For example for the WordPress Popular Posts widget the URL is https://guides.wp-bullet.com/wp-admin/admin-ajax.php?action=wpp_get_recently&id=5. You can then target it like this in sub vcl_recv for my example Popular Posts URL by doing a regular expression match, here the ~ means ‘contains’.

##Possibility to cache admin-ajax GET requests
  
 if ((req.url ~ "wpp_get_recently") && !req.http.cookie ~ "wordpress_logged_in" ) {
    return (hash);
  }

In this Varnish snippet from sub vcl_backend_fetch we are specifying how long Varnish should store the cache for in days, you can use h for hours or m for minutes instead if you prefer. This is setting the TTL (time to live) for the cached URL. We also set the beresp.grace period for the AJAX GET requests so if Varnish can’t contact the backend to get fresh data after the beresp.ttl expires, it will keep delivering the old data for another day or until the backend becomes available.

if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

Similarly to the other snippet you can target the specific URL instead

if ((req.url ~ "wpp_get_recently") && !req.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

Test your Varnish configuration has valid syntax

varnishd -C -f /etc/varnish/default.vcl

Reload the Varnish configuration

sudo sevice varnish reload

Verify Varnish is Caching the AJAX GET Requests

If you have the X-Cache header set you can use curl to grab the AJAXified URL. You can get the full URL of the AJAX request by using something like Firebug in Firefox or the Developer tools in Chrome.

sudo apt-get install curl -y

Now you can curl the URL of the AJAX request, the I switch outputs the response headers. You should do this at least twice to ensure there is a HIT.

curl -I "https://guides.wp-bullet.com/wp-admin/admin-ajax.php?action=wpp_get_recently&id=5"

The output should look something like this, notice the X-Cache header is reporting a HIT.

HTTP/1.1 200 OK
Date: Thu, 08 Sep 2016 13:05:25 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d2e66eba2f781777511608736f1e39aed1473339925; expires=Fri, 08-Sep-17 13:05:25 GMT; path=/; domain=.wp-bullet.com; HttpOnly
X-Robots-Tag: noindex
X-Content-Type-Options: nosniff
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Varnish: 196648 196646
Age: 47
Via: 1.1 varnish-v4
Access-Control-Allow-Origin: https://guides.wp-bullet.com
X-Cache: HIT
Server: cloudflare-nginx
CF-RAY: 2df29ca63b1b2c3c-AMS

You can use Varnishlog to see what is going on as well and how the GET requests are being cached

varnishlog

You can see the requests are now cached and showing a HIT status

* << Request >> 196648
- Begin req 196647 rxreq
- Timestamp Start: 1473339925.145109 0.000000 0.000000
- Timestamp Req: 1473339925.145109 0.000000 0.000000
- ReqStart 141.101.105.55 17500
- ReqMethod GET
- ReqURL /wp-admin/admin-ajax.php?action=get_recently&widget_id=5
- ReqProtocol HTTP/1.1
- ReqHeader Host: guides.wp-bullet.com
- ReqHeader Connection: Keep-Alive
- ReqHeader Accept-Encoding: gzip
- ReqHeader CF-IPCountry: NL
- ReqHeader X-Forwarded-For: 178.62.158.10
- ReqHeader CF-RAY: 2df29ca63b1b2c3c-AMS
- ReqHeader X-Forwarded-Proto: http
- ReqHeader CF-Visitor: {"scheme":"http"}
- ReqHeader User-Agent: curl/7.26.0
- ReqHeader Accept: */*
- ReqHeader CF-Connecting-IP: 188.63.158.10
- ReqUnset X-Forwarded-For: 188.63.158.10
- ReqHeader X-Forwarded-For: 188.63.158.10, 141.101.105.55
- VCL_call RECV
- ReqHeader X-Actual-IP: 188.63.158.10
- ReqUnset X-Forwarded-For: 188.63.158.10, 141.101.105.55
- ReqHeader X-Forwarded-For: 178.63.158.10, 141.101.105.55, 141.101.105.55
- ReqHeader Cookie:
- ReqUnset Cookie:
- ReqHeader Cookie:
- ReqUnset Accept-Encoding: gzip
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- Hit 196646
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Server: nginx
- RespHeader Date: Thu, 08 Sep 2016 13:04:38 GMT
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader X-Robots-Tag: noindex
- RespHeader X-Content-Type-Options: nosniff
- RespHeader Expires: Wed, 11 Jan 1984 05:00:00 GMT
- RespHeader Cache-Control: no-cache, must-revalidate, max-age=0
- RespHeader X-Frame-Options: SAMEORIGIN
- RespHeader Content-Encoding: gzip
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 196648 196646
- RespHeader Age: 47
- RespHeader Via: 1.1 varnish-v4
- VCL_call DELIVER
- RespUnset Server: nginx
- RespHeader Access-Control-Allow-Origin: https://guides.wp-bullet.com
- RespHeader X-Cache: HIT
- VCL_return deliver

Now you have successfully cached WordPress admin-ajax GET requests with Varnish 4 and your AJAXified content will load faster while PHP and MySQL get a much deserved rest.

1 thought on “How to Cache AJAX GET Requests with Varnish 4”

  1. Hello,

    great post! Do you have figured out a full .vcl config for varnish 4 and woocommerce?

Comments are closed.