Comments on: 7 More Good Tips on Logging

By: stephan

stephan — Tue, 28 Apr 2009 13:18:48 +0000

In the book mentioned, Spread is used to log all messages to, and then add listeners which will either write to file or monitor special events. Those are easier to write than log analyzers that issue actions (so well it is kind of application-to-application messageing).

Looking at the Second Life Message Queue Evaluation

http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

they use RabbitMQ for this:
“RabbitMQ is already installed in our image and is currently being deployed by the data warehousing team (Ivan in particular) to handle syslog message delivery.”

By: Andrew

Andrew — Tue, 28 Apr 2009 12:41:50 +0000

Hey Stephan,

I was reviewing the Spread toolkit and am curious as to how you think it would be used for something other than application-to-application messaging. When I saw the link, I thought, “Wow, there might be an alternative to Syslog that supports larger messages and could be a log event retainer!”

Thoughts?

By: Ridhav

Ridhav — Wed, 01 Apr 2009 04:47:23 +0000

Its very nice and informative article.
@Michael It does decrease the performance after enabling debug in production in most of the cases even after load balancing.
But as debug is very much required to trace the errors, we can have a replica of prod deployed some where as pre-prod which points to same database and is debug enabled where we can leave the prod server to be just error enabled. so the developers can easily replicate the problems there on pre-prod without compromising the performance of production server.

By: french_c

french_c — Thu, 12 Mar 2009 22:05:43 +0000

Should have read the comments though;)

“(Beside one-liners for easier grepping and awking)”

By: french_c

french_c — Thu, 12 Mar 2009 22:04:46 +0000

Seems I have to add another important rule: Excluding stack traces never ever use more than one line to log. In other words: Log statements may not contain an CR/LF – especially in production.

I personally don’t like the error code approach. While it sounds reasonable it becomes tricky in large scale applications. And what do you do with all our 3rd party libraries that don’t follow this rule?

By: Johan Karlsson

Johan Karlsson — Thu, 12 Mar 2009 08:28:18 +0000

Hi,

your articles about logging are very good. I wished I had written them :)

I have written an article with my personal comments about your article:

http://myossdevblog.blogspot.com/2009/03/how-to-do-logging.html

Best Regards
Johan

By: Logging by Schauderhaftes

Logging by Schauderhaftes — Tue, 24 Feb 2009 20:52:50 +0000

[...] Schmidt writes about logging in his latest article. I think he has 7 mostly valid points and even 7 more in an older post, but there is one advice [...]

By: stephan

stephan — Tue, 24 Feb 2009 15:24:03 +0000

Splunk is excellent – a little pricy for smaller companies though.

By: Banjollity

Banjollity — Tue, 24 Feb 2009 12:30:28 +0000

Google “splunk” – an excellent log management tool.

By: Kirk

Kirk — Tue, 24 Feb 2009 09:58:21 +0000

@Kirk: What does machine readable mean? (Beside one-liners for easier grepping and awking)

it means create logs that are easily parsed, greped or awked.

But if you do not know what you are looking for, machine readable doesn’t help, you need human readable.

right, but I often know what I’m not looking for ;-)

By: stephan

stephan — Tue, 24 Feb 2009 08:37:40 +0000

@Kirk: What does machine readable mean? (Beside one-liners for easier grepping and awking)

But if you do not know what you are looking for, machine readable doesn’t help, you need human readable.

By: Kirk

Kirk — Tue, 24 Feb 2009 08:10:08 +0000

Make the logs machine readable. Programs a much better at looking at massive amounts of data than I am.

By: stephan

stephan — Tue, 24 Feb 2009 06:07:28 +0000

@ Dimitris: Thanks, appreciated.

By: Dimitris Andreou

Dimitris Andreou — Tue, 24 Feb 2009 02:55:22 +0000

(a unique, a university, not an unique, or an university. sorry for the pedantry)

By: stephan

stephan — Mon, 23 Feb 2009 14:59:37 +0000

@Michael: What I’ve learned is that you should be able to turn everything on or off in production, this includes logging.

I find it more useful to have lots of monitoring points in production to find problems than debug statements with logging.

” I’m sure it does, sometimes, in some situations, but this is something that needs to be benchmarked.”

Surely it depends on the debug logging culture (how many, what, who expensive are the logged expressions) in a company on how much it slows down your application. But I’ve seen 30%-50% performance decrease due to turned on debugging.

“You gotta test it.”

Yes, as always. But as a rule of thumb, better have it turned off.

By: Michael Campbell

Michael Campbell — Mon, 23 Feb 2009 14:20:21 +0000

I think a refinement of #1 is, “In a production environment, ensure a way to turn logging on and off.” When debugging a production issue, having debug and/or trace level logs can be invaluable.

Also, much is made of debug logs “slowing down apps”. I’m sure it does, sometimes, in some situations, but this is something that needs to be benchmarked. I’ve seen apps that produce a few hundred megs of debug logs per second that when turned down… had no statistically significant performance gain. This is particularly true with widely distributed infrastructures. If everything is on one box, the story could be very different. The point is, there is no “best practice” here in terms of performance. You gotta test it.

By: stephan

stephan — Mon, 23 Feb 2009 14:03:43 +0000

I’d say it’s very important to always log a session/user id to find corresponding entries in different logs (especially when crossing boundaries client< ->server, tomcat< ->RMI backend etc.)

Reporting: Call a backend with REST and log the event

If you want to log client exceptions (and reporting), it’s best to not log every event indivudually but collect events and say post 10 events to the backend. Most often it’s not that important when losing frontend exceptions compared to server load.

By: Mladen Miljic

Mladen Miljic — Mon, 23 Feb 2009 13:14:10 +0000

Hi Stephan,
thanks for the hints. Can you say something about client side loging? How to log Exceptions occured in the browser (for example) and bind them to a corespending Exception on the server? (Ajax, DOM manipulating, Reporting…)