Invisible Race Conditions: The Cache Has Poisoned Us

We got a memory corruption error one day that was quite interesting. It was in a place where we had previously fixed a memory corruption error and was, at a glance, quite impossible.
The code would checkout an item from the cache and increment its ref count, which will keep it alive for as long as we were using it. But something made it fail, and quite horribly, too. We finally tracked the code down to this piece of code, which is run when we update the cache: