How to Cache AJAX GET Requests with Varnish 3

I love caching with Varnish. Its vcl (Varnish Control Language) allows me to be precise about what and how I want to cache specific URLs in WordPress and WooCommerce. One area which can be a pain for performance enthusiasts is showing up-to-date recent posts in a sidebar widget or show off your popular posts in the footer. The traditional way to work around this is by using AJAX so javascript makes a call for PHP to pull fresh data out of the MySQL database. This is very convenient but also potentially resource heavy and slow because of the entire process flow. Let's cache those instead with Varnish.

The Varnish 4 version of this tutorial can be found here.

Speed Benefits of Caching AJAX Get Requests with Varnish

When I AJAXify the Recently and WordPress Popular Posts widget it takes between 200-300ms for the site to display the data. That is not the plugins' fault, it is just the nature of AJAX (technical explanation here). Luckily we can cache this AJAX request and get it down to 50ms with Varnish 3 – a Varnish 4 equivalent will be published as well. I will assume you already have Varnish 3 installed and configured on your VPS or dedicated server.

Here are some pingdom results before and after caching the AJAX requests.

Uncached AJAX Response Time

Using the WordPress Popular Posts AJAXified widget without Varnish took 141 ms.

Varnish cached AJAX Response Time

Using the WordPress Popular Posts AJAXified with Varnish took 68 ms.

That is over a 50% increase in performance by caching the AJAX request with Varnish.

How to Cache AJAX GET Requests with Varnish 3

First let's understand how AJAX requests work.

How AJAX GET Requests Work in WordPress

The diagram Regular AJAX GET Request with WordPress shows how many steps are involved in a typical AJAX request in WordPress.

regular-ajax-get-request-wordpress

The diagram Varnish cached AJAX GET Request with WordPress shows how much faster it is to serve AJAX GET requests with Varnish

varnish-cache-ajax-get-request-wordpress

Cache AJAX GET Requests using Varnish 3

Open up your Varnish vcl file

sudo nano /etc/varnish/default.vcl

Add the red sections which are carefully placed in the vcl to not conflict with the vcl order. First we need to catch the AJAX requests for non-logged in users before wp-admin requests are passed and then set the time to live in sub vcl_fetch for our captured and cached AJAX GET requests. The snippets are explained in the next section.

...

sub vcl_recv {

...

  # PIPE ALL NON-STANDARD REQUESTS
  # ##########################################################
  if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
      return (pipe);
  }

  # ONLY CACHE GET AND HEAD REQUESTS
  # ##########################################################
  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

##Possibility to cache admin-ajax GET requests
  
 if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    return (lookup);
  }

  # OPTIONAL: DO NOT CACHE LOGGED IN USERS (THIS OCCURS IN FETCH TOO, EITHER
  # COMMENT OR UNCOMMENT BOTH
  # ##########################################################
  if ( req.http.cookie ~ "wordpress_logged_in" ) {
    return( pass );
  }


  # IF THE REQUEST IS NOT FOR A PREVIEW, WP-ADMIN OR WP-LOGIN
  # THEN UNSET THE COOKIES
  # ##########################################################
  if (!(req.url ~ "wp-(login|admin)") 
    && !(req.url ~ "&preview=true" ) 
  ){
    unset req.http.cookie;
  }


  # IF BASIC AUTH IS ON THEN DO NOT CACHE
  # ##########################################################
  if (req.http.Authorization || req.http.Cookie) {
    return (pass);
  }
  
  # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED
  # ##########################################################
  return (lookup);
}

# HIT FUNCTION
# ##########################################################
sub vcl_hit {
  # IF THIS IS A PURGE REQUEST THEN DO THE PURGE
  # ##########################################################
  if (req.request == "PURGE") {
    purge;
    error 200 "Purged.";
  }
  return (deliver);
}

# MISS FUNCTION
# ##########################################################
sub vcl_miss {
  if (req.request == "PURGE") {
    purge;
    error 200 "Purged.";
  }
  return (fetch);
}

# FETCH FUNCTION
# ##########################################################
sub vcl_fetch {
  # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC 
  # TENDANCY TO SET VARY USER-AGENT.  YOU MAY OR MAY NOT WANT
  # TO DO THIS
  # ##########################################################
  set beresp.http.Vary = "Accept-Encoding";

  # IF NOT WP-ADMIN THEN UNSET COOKIES AND SET THE AMOUNT OF 
  # TIME THIS PAGE WILL STAY CACHED (TTL)
  # ##########################################################

#set the length of time to cache ajax GET requests

if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

##set how long to cache for anything that is not admin or ajax

  if (!(req.url ~ "wp-(login|admin)|admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in") {
    unset beresp.http.set-cookie;
    set beresp.ttl = 52w;
    set beresp.grace = 1w;
  }

  if (beresp.ttl <= 0s ||
    beresp.http.Set-Cookie ||
    beresp.http.Vary == "*") {
      set beresp.ttl = 120 s;
      return (hit_for_pass);
  }

  return (deliver);
}

.....

Save the configuration with Ctrl+X, Y and Enter, don't restart Varnish yet.

Caching AJAX Requests with Varnish Explained

In this Varnish snippet from sub vcl_recv section we are catching AJAX requests for non-logged in users and telling Varnish to look them up in its cache. It is important this snippet comes before you pass wp-admin in Varnish!

##Possibility to cache admin-ajax GET requests
  
 if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    return (lookup);
  }

If you want to be even more specific by finding the exact AJAX URL in Chrome's Developer tools or Firefox Firebug. For example for the WordPress Popular Posts widget the URL is https://guides.wp-bullet.com/wp-admin/admin-ajax.php?action=wpp_get_popular&id=2. You can then target it like this in sub vcl_recv for my example Popular Posts URL by doing a regular expression match, here the ~ means ‘contains'.

##Possibility to cache admin-ajax GET requests
  
 if ((req.url ~ "wpp_get_popular") && !req.http.cookie ~ "wordpress_logged_in" ) {
    return (lookup);
  }

In this Varnish snippet from sub vcl_fetch we are specifying how long Varnish should store the cache for in days, you can use h for hours or m for minutes instead if you prefer. This is setting the TTL (time to live) for the cached URL. We also set the beresp.grace period for the AJAX GET requests so if Varnish can't contact the backend to get fresh data after the beresp.ttl expires, it will keep delivering the old data for another day or until the backend becomes available.

if ((req.url ~ "admin-ajax.php") && !req.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

Similarly to the other snippet you can target the specific URL instead, here wpp_get_popular

if ((req.url ~ "wpp_get_popular") && !req.http.cookie ~ "wordpress_logged_in" ) {
    unset beresp.http.set-cookie;
    set beresp.ttl = 1d;
    set beresp.grace = 1d;
  }

Test your Varnish configuration is working

varnishd -C -f /etc/varnish/default.vcl

Reload the Varnish configuration

sudo sevice varnish reload

Verify Varnish is Caching the AJAX GET Requests

If you have the X-Cache header set you can use curl to grab the AJAXified URL. You can get the full URL of the AJAX request by using something like Firebug in Firefox or the Developer tools in Chrome.

sudo apt-get install curl -y

Now you can curl the URL of the AJAX request, the -I switch outputs the response headers. You should do this at least twice to ensure there is a HIT.

curl -I "https://guides.wp-bullet.com/wp-admin/admin-ajax.php?action=wpp_get_popular&id=2"

The output should look something like this, notice the X-Cache header is reporting a HIT.

HTTP/1.1 200 OK
Date: Mon, 23 May 2016 10:49:42 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d5d000d0a8fe7915564b4de8bb5874cff1464000582; expires=Tue, 23-May-17 10:49:42 GMT; path=/; domain=.wp-bullet.com; HttpOnly
X-Robots-Tag: noindex
X-Content-Type-Options: nosniff
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Varnish: 1694929555 1694929352
Age: 214
Via: 1.1 varnish
X-Cache: HIT
Server: cloudflare-nginx
CF-RAY: 2a77f158e1ff3470-LHR

You can use Varnishlog to see what is going on as well and how the GET requests are being cached

varnishlog

You can see the requests are now cached and showing a HIT status

   29 SessionOpen  c 188.114.103.205 14908 :80
   29 ReqStart     c 188.114.103.205 14908 1694930274
   29 RxRequest    c GET
   29 RxURL        c /wp-admin/admin-ajax.php?action=wpp_get_popular&id=2
   29 RxProtocol   c HTTP/1.1
   29 RxHeader     c Host: guides.wp-bullet.com
   29 RxHeader     c Connection: Keep-Alive
   29 RxHeader     c Accept-Encoding: gzip
   29 RxHeader     c CF-IPCountry: IT
   29 RxHeader     c X-Forwarded-For: 87.8.64.231
   29 RxHeader     c CF-RAY: 2a7801fe52104322-MXP
   29 RxHeader     c X-Forwarded-Proto: https
   29 RxHeader     c CF-Visitor: {"scheme":"http"}
   29 RxHeader     c accept: */*
   29 RxHeader     c origin: https://guides.wp-bullet.com
   29 RxHeader     c user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/601.5.17 (KHTML, like Gecko) Version/9.1 Safari/601.5.17
   29 RxHeader     c accept-language: it-it
   29 RxHeader     c referer: https://guides.wp-bullet.com/
   29 RxHeader     c CF-Connecting-IP: 87.8.64.231
   29 VCL_call     c recv
   29 VCL_acl      c NO_MATCH purge
   29 VCL_return   c lookup
   29 VCL_call     c hash
   29 Hash         c /wp-admin/admin-ajax.php?action=wpp_get_popular&id=2
   29 Hash         c guides.wp-bullet.com
   29 VCL_return   c hash
   29 Hit          c 1694716489
   29 VCL_call     c hit deliver
   29 VCL_call     c deliver deliver
   29 TxProtocol   c HTTP/1.1
   29 TxStatus     c 200
   29 TxResponse   c OK
   29 TxHeader     c Content-Type: text/html; charset=UTF-8
   29 TxHeader     c Access-Control-Allow-Credentials: true
   29 TxHeader     c X-Robots-Tag: noindex
   29 TxHeader     c X-Content-Type-Options: nosniff
   29 TxHeader     c Expires: Wed, 11 Jan 1984 05:00:00 GMT
   29 TxHeader     c Cache-Control: no-cache, must-revalidate, max-age=0
   29 TxHeader     c Pragma: no-cache
   29 TxHeader     c X-Frame-Options: SAMEORIGIN
   29 TxHeader     c Content-Encoding: gzip
   29 TxHeader     c Vary: Accept-Encoding
   29 TxHeader     c Content-Length: 663
   29 TxHeader     c Accept-Ranges: bytes
   29 TxHeader     c Date: Mon, 23 May 2016 11:01:04 GMT
   29 TxHeader     c X-Varnish: 1694930274 1694716489
   29 TxHeader     c Age: 52866
   29 TxHeader     c Via: 1.1 varnish
   29 TxHeader     c Connection: keep-alive
   29 TxHeader     c X-Cache: HIT
   29 Length       c 663
   29 ReqEnd       c 1694930274 1464001264.969143867 1464001264.969459534 0.000077724 0.000202417 0.000113263

Now you have successfully cached WordPress admin-ajax GET requests with Varnish 3 and your AJAXified content will load faster while PHP and MySQL get a much deserved rest.