What is caching?

Caching is basically storing copies of data in a temporary, high-speed storage layer so that future requests for that data can be served much faster. It only works because programs tend to access the same data repeatedly. This predictable behavior comes in two scenarios:

1. If a piece of data is accessed once, it is highly likely to be accessed again soon. (e.g., a user refreshing their profile page).
2. If a piece of data is accessed, data stored physically close to it is likely to be accessed soon. (e.g., looping through an array or reading a sequential file).

I’ve used caching to speed up some apps way beyond however, I view it as more of a second step after optimizing database queries when considering improving application speeds.

Cache strategies

Application layer caching

This is whereby the caching happens deep in the app layer ie close to the database. An example would be an algorithm checking for popular user profiles to cache their content(like follower count) since we can pre-empt that their profiles will be visited alot so we can lessen the load on the database. The algorithm is part of the application but sits closer to the database than the application users.

Request level caching

This is whereby the caching happens in the request typically for requests involving parameters like paginations eg /users?page=1 - pagination involves read heavy APIs, so consider a scenario of loading followers of a popular user profile, at the very least they are ordered by time they followed but loading all at once may be really expensive so we can paginate the results and store them in cache.

Conditional caching

This is whereby a cache is created only when certain conditions match. An example would be to use an “e-tag” and “last modified” headers when server sends back a response, the next time a client sends a request it sends with its last e-tag if the e-tag header matches the server responds with a 304 status code otherwise it returns a new e-tag and last modified time with a 200 status code response.

Cache invalidation

This is the process by which cache data is removed or marked as not usable. Using the example of popular user profiles their cache for follower count might need to be invalidated every few hours as they gain new followers so as not to show stale data. Some invalidation strategies to reconsider are:

Write-Through Invalidation

This technique is whereby the application writes to the cache as well as database simultaneously. The write automatically invalidates older/stale data. It has the advantage of ensuring a fresh cache always when reading but the trade off is that the writes are slow.

Write-Behind

This technique involves the use of a separate process usually running in the background after a database write to update the cache and invalidate old data. The database remains the source of truth for the data and it makes the writes faster but introduces the possibility of stale cache data and increased implementation complexity

TTL

In this approach the cache uses a TTL(time to live) key which determines when it expires and not to be used. It is best for data with a predictable expiration for example product pricing during a holiday season.

Some gotchas with caching

Well as with most stuff, this has some trade offs. To list some of the scenarios:

Cache having a stampede(Thundering herd)

Typically caches have a set expiry time, when this expiry time reaches for many of the caches at once all requests are routed to the main database which may overwhelm the database.

We may employ a couple techniques to resolve this:

1. Locking - each request for that specific expired cache key attempts to acquire a lock to recompute the expired cache, if the lock isn’t acquired there a couple workarounds:
2. Request waits till the cache key is recomputed elsewhere
3. Request returns a not-found and client handles a retry
4. System maintains stale cache version while new value is computed
5. The locking method though needs an additional write implementation for the lock itself which maybe fairly advanced
6. Offload cache recomputation to an external process - this process can activate proactively by checking if a cache key is nearing its expiry or reactively by activating when there is a cache miss
7. Probablistic early recomputation - this features a request having a probability to trigger cache recomputation, this staggers the expiry and minimizes the “stampede” issue

Cache penetration

This is whereby there is a cache miss and the data doesn’t exist in db either. This then leads to a cycle where the cache system is hit as well as the database which leads to unnecessary load as the system tries to retrieve non-existent data. This can destabilise the system if the request volume is high

To resolve this:

1. Implementing a cache value for non-existent keys so that follow up requests hit the cache instead of pointlessly hitting the DB
2. Adding a quick look up key for each entry in DB, so that if key is absent the record is known to not exist early this is called a bloom filter

Cache crash

This is the scenario when the cache system fails entirely, so with no cache layer every single request hits the database which then again might overwhelm it making system unstable

Some couple of ways to tackle this:

1. Circuit breaking - when a portion of the system is down temporarily block incoming requests to prevent system overload to prevent total system crash
2. Deploying highly available redundant clusters to minimize the impact of one cache crashing
3. Cache pre-warming to populate cache before it is put into service

To close caching isn’t a silver bullet. Try thinking of where the bottle neck is ie is it in rendering or the actual query to appropriately use it and get the best result

Resources

https://planetscale.com/blog/distributed-caching-systems-and-mysql