Login

Another Redis use case: Centralized logging

Log analysis has become a difficult task in our production environment at work because logs are distributed on different machines and in different files. So, we wanted all the exception logs from all of our apps to be tracked centrally and viewed in single console. 

And once again, we found another good use case for Redis. Our strategy is to dump all of critical logs in to a Redis List and have a background worker which continuously pulls logs from the Redis List and write stuff in log file.

As we use python for all our backend work, I quickly wrote a Log Handler that can dump log messages in to Redis. 

So RedisLogHandler class looks like this:

To hook up this RedisLogHandler to your application logger, all you need to do is following:

I quickly hooked it up to our all of our background jobs in celery and I can see it working. Let me know what you think of this solution.

8 responses

Jay IVR upvoted this
David Moreno said
How many events are you planning to process using this?
Mikhail P said
Please read this http://urbanairship.com/blog/2010/10/05/centralized-logging-using-rsyslog/ for a vastly superior, actually scalable approach to this usecase.

Second result on google for "centralized logging."

bluesmoon said
Interesting. I've been thinking of using zeromq in a similar fashion. Use a 0MQ pub socket to send out log messages from all boxes, and a central SUB socket to receive them. We could also set up two SUBs on two different boxes for BCP in case one of the boxes died.
j. random websurfer said
> Can't do much here--probably redis have stopped responding...

This is unreliable. Logs should be stored locally until they're either expired or successfully transferred to a remote storage.

Unfortunately, AFAIK Redis doesn't support many-to-one replication, so you can't have a local Redis instances replicating to a central one. A rsyslogd or, if one feels adventurous, something like a RabbitMQ cluster would solve the task.

mattbillenstein said
+1 for rsyslog -- use the right tool for the job...

And specifically for capturing exceptions, we use a simple little python server to collect and aggregate exceptions. When an exception occurs on one of our application servers, we have an exception handler which http posts that exception to an internal central server which aggregates, counts, sorts, and displays exceptions in a simple web page.

This allows us to see exceptions in real-time without having to wade through the raw server logs...

richard bucker said
I would like to add your code to my project... but there is no licensing in the doc. Please take a look and provide some comment?

https://github.com/rbucker881/sub-watcher
http://sub-watcher.com
http://richardbucker.com

Thanks

Dylan said
Check out the Logstash project it does all this and much more ! Plus its free