Aerospike: A Misconfiguration Story

Introduction

Aerospike is an in-memory NoSQL database that is optimized for fast lookup, which makes it a great tool for high-performance systems. This was particularly attractive to us as our servers deal with a very high volume of request (around 300,000 per second) and each of these has to be answered in a limited time (<50 ms). Thus, data retrieval times are vital to the continuous operation of our system, which prompted us to use Aerospike as a distributed persisted cache for certain parts of our system.

Problem

As with any new technology, there are certain pitfalls when using Aerospike with default settings. Initially, we saw a noticeable speed improvement that, along with the ease of use, made it seem like we had found the perfect solution out of the box. This was until we noticed an incongruity within our records.

Further investigation revealed that some of our records were missing. There seemed to be no common attribute in the removed records, so a bug in our record keeping code seemed unlikely.

Cause

When we started looking at our Aerospike dashboard, we noticed a breach of the RAM high watermark. Additionally, we noticed a metric that we never paid any particular attention to before. The ‘Evicted Objects’ counter.

It turned out what was causing the missing data was Aerospike evicting data to free RAM in order to retain its failover integrity. Eviction is a standard mechanism, triggered by the RAM high watermark breach. This process removes records to free memory for as long as the high watermark remains breached. To choose which records to evict first, it only considers time-to-live (TTL).

The apparent randomness we saw was a result of Aerospike grouping records in 100 buckets over which the maximum TTL is evenly distributed (in our case, this resulted in a width of around 80000 seconds, which is a little less than a day) and starting to delete from the first bucket, then the second, and so on. Unfortunately, as we use Aerospike as a cache, some of our most vital data has a TTL of roughly a day. This meant that that data almost entirely landed in the first bucket and was therefore viable for removal in random order.

Additionally, a large portion of our non-vital data has a much larger TTL of several weeks, resulting in the first buckets mainly containing vital data. As a result, we experienced the removal of our most important data in random order.

It is also worth noting that if the high watermark is breached without any evictable records, Aerospike will go into stop-write and only be able to read data until memory is freed.

Solution

While the eviction mechanism is crucial for the operation of Aerospike with limited resources, we need to instruct the NoSQL database how to treat different sets so we don’t lose important data. We found out that when simply displaying Aerospike sets in your shell, you get some extra information, including what looks like the following.

Fig. 1. disable-eviction setting for sets in Aerospike shell

Regrettably, we never paid enough attention to such settings. Turns out, ignoring things that “have always been this way” might not always be the best idea. The disable-eviction setting was, in fact, a very efficient way to solve our problem (apart from freeing RAM space). By enabling it you prevent data from certain sets from being removed in the eviction process. There is a quite convenient way to change this setting dynamically, without the need to restart the Aerospike nodes:

Fig. 2. Command to disable eviction for distinct sets

After enabling this setting on our key sets, no more critical records were influenced by the eviction process and we could start focussing on freeing some of the RAM.

Fig. 3. disable-eviction setting, enabled for vital sets

For a long term solution, this setting should also be statically stored in the aerospike.conf config file, otherwise it will not remain upon restarting the node.

Conclusion

As we described earlier, Aerospike’s eviction process resulted in a unexpected loss of data in our system. The problem, though, was not the eviction itself. That process is important for Aerospike to maintain a good performance when not all nodes within a cluster are available. The real problem was misconfiguration. We had been using the default settings and had not bothered to dive deeper into the technology so long as it did what we wanted it to do smoothly. What we learned is that, when using new technologies, we should not take the default settings for granted, even if they seem to work out of the box. Instead, we need to learn what impact even apparently unimportant settings may have.

References

Post 3.7 changes

Starting with aerospike version 3.8. Two major changes have been introduced for eviction:

For one, the number of buckets is now configurable to anything between 100 and 10 million.
Also records from buckets do not get randomly removed anymore, instead the entire bucket gets dropped at once

While updating our aerospike version might have been a good idea in general, in this case it would probably not have helped a whole lot, since even with a larger bucket size our important data would have been removed first.