Another Redis use case: Centralized logging
Log analysis has become a difficult task in our production environment at work because logs are distributed on different machines and in different files. So, we wanted all the exception logs from all of our apps to be tracked centrally and viewed in single console.
And once again, we found another good use case for Redis. Our strategy is to dump all of critical logs in to a Redis List and have a background worker which continuously pulls logs from the Redis List and write stuff in log file.
As we use python for all our backend work, I quickly wrote a Log Handler that can dump log messages in to Redis.
So RedisLogHandler class looks like this:
To hook up this RedisLogHandler to your application logger, all you need to do is following:
I quickly hooked it up to our all of our background jobs in celery and I can see it working. Let me know what you think of this solution.
Second result on google for "centralized logging."
This is unreliable. Logs should be stored locally until they're either expired or successfully transferred to a remote storage.
Unfortunately, AFAIK Redis doesn't support many-to-one replication, so you can't have a local Redis instances replicating to a central one. A rsyslogd or, if one feels adventurous, something like a RabbitMQ cluster would solve the task.
And specifically for capturing exceptions, we use a simple little python server to collect and aggregate exceptions. When an exception occurs on one of our application servers, we have an exception handler which http posts that exception to an internal central server which aggregates, counts, sorts, and displays exceptions in a simple web page.
This allows us to see exceptions in real-time without having to wade through the raw server logs...
https://github.com/rbucker881/sub-watcher
http://sub-watcher.com
http://richardbucker.com
Thanks