Once a particular event of mine hits, I want to be able to expire the entire contents of an Ohm based object in my Ruby on Rails app. Is it possible to tie in Redis + expire to do this currently? Objects have multiple keys associated with them when using Ohm, including indices etc. I want to ensure that everything gets properly cleaned - that's why I was wondering if there is an officially supported way of doing this.
Unfortunately, no. There have been several attempts to solve this puzzle, but every one that I have seen leaves artifacts in the Ohm dataset. None of them work with unique attributes, and to my knowledge all of them leave Ohm data in an inconsistent state.
Some examples:
For instance, when an Ohm model saves, there are several fields added to Redis hashes and also members added to Redis sets. While you can set an expiry on an entire hash or set, you cannot expire a single field of a Redis hash, or a single member of a set. Either the whole hash or set expires.
Here's the main problem: If these sets and hashes were to expire, you would lose your entire index on the model, or your complete record of unique attributes. So a common problem when using any of the Ohm expire mixins is that even though the main data key expires, a call to find will still return a record from the index, but with a nil hash value. And if you have a unique attribute specified in your model, you can never again call create on that model with the expired value without raising an exception, even though the data itself has expired.
There are no expiry callbacks in Redis, so there's no way to trigger removing hash fields or set members when a particular key expires. Several people have requested allowing hash fields or set members to have TTLs on the Redis issues list, but they have all been (quite reasonably) closed with answers such as this:
Hi, this will not be implemented by original design:
No nested types in keys. No complex features in single fields of aggregate data types. The reasoning is much more complex and is a lot biased by personal feelings, preference and sensibility so there is no objective way I can show you this is better ;)
Closing. Thanks for reporting.
For example, here are some comments from the Ohm source code (ohm.rb, 651-699):
# The base class for all your models. In order to better understand
# it, here is a semi-realtime explanation of the details involved
# when creating a User instance.
#
# Example:
#
# class User < Ohm::Model
# attribute :name
# index :name
#
# attribute :email
# unique :email
#
# counter :points
#
# set :posts, :Post
# end
#
# u = User.create(:name => "John", :email => "foo@bar.com")
# u.incr :points
# u.posts.add(Post.create)
#
# When you execute `User.create(...)`, you run the following Redis
# commands:
#
# # Generate an ID
# INCR User:id
#
# # Add the newly generated ID, (let's assume the ID is 1).
# SADD User:all 1
#
# # Store the unique index
# HSET User:uniques:email foo@bar.com 1
#
# # Store the name index
# SADD User:indices:name:John 1
#
# # Store the HASH
# HMSET User:1 name John email foo@bar.com
#
# Next we increment points:
#
# HINCR User:1:counters points 1
#
# And then we add a Post to the `posts` set.
# (For brevity, let's assume the Post created has an ID of 1).
#
# SADD User:1:posts 1
#
But the way people generally try to expire Ohm data is with something much simpler like this (no manipulation of uniques or model-wide indices):
Ohm.redis.expire(object.key, @expire)
Ohm.redis.expire("#{object.key}:_indices", @expire)
In summary, the best way to get fine-grained control of data expiry in Redis is to design your own storage methodologies using a low-level interface such as redis-rb.
Rather than expiring using Redis, a more reliable way to expire Ohm objects is to examine and delete the objects directly, for example by having separate expiry thread. This can check model objects, detect those that have expired, and request deletion via Ohm.
Ohm performs deletions atomically using a Lua script, this correctly cleans up any keys associated with the object, and removes references to the object from other Ohm structures. Using Ohm deletion is your best option for clean expiry.
By using such a design, you may need to trade off performance for reliability: your program will essentially be performing a Garbage Collection across all known objects, which may involve extensive memory access and/or Redis queries.
Here is an example of how a clean up thread can work:
def start_expiry_thread expiry_cycle
Thread.new {
while (true) do
MyOhmModel.all.each do |object|
object.delete if object.expired
end
sleep expiry_cycle
end
}
end
Where the expired function can be implemented along the following lines in your Ohm model class (via inheritance or mixin), with support from the Ohm::Timestamps module:
@max_life = 3600
def expired
(time_since_update > @max_life)
end
def time_since_update
updated_secs = self.updated_at.to_i
(Time.now.to_i - updated_secs)
end
Notes:
Take care when introducing threading - use locking to prevent simultaneous access to the same object by two threads, or simply avoid shared objects and get your expiry thread to go directly to Redis.
If your application process has a regular scheduled restart, an alternative is to perform clean up during the startup process, before any other worker threads are started.
Make sure your other application code handles expiry gracefully - the underlying Redis objects will be gone, which means any equivalent objects held by application code will be invalid.
The above is a simplified version of code from a production system, so bugs may have crept in during simplification.
Also more generally speaking: please check that you really need Ohm. You should be thinking hard about the technical design of your system if you're considering adding what amounts to a relational layer on top of a key-value store.
I've developed a gem for Ohm that takes care of it.
Take a look at: ohm-expire