the blog for developers

7 More Good Tips on Logging

Logging in web applications is important – to know what’s going on, for performance tuning and incident analyis. This is my second post about logging. The first post “7 Good Rules to Log Exceptions” was specific to logging exceptions, ths is about logging in general. What makes your logs more useful to you?

Nerdy Bookshelf Part One
Creative Commons Licensecredit: schoschie

1. No debugging logs in production

I have seen time and again that debug logging is enabled in production. This can be intentional or happening by some developers who accidently checked in a debugging logging configuration. Enabled debugging slows down your application remarkedly and makes it impossible to read production logs due to noise. Make sure during deployments – best with some scripts – that debugging level logging is disabled during production.

2. Look through your logs

Some companies have good logging in their production system, but do not look into their logs. Look into your logs, discover issues (bugs, performance, memory) with your application and fix them. Essentially your logs should be without known errors.

3. Log to the correct log level

Developers who write logging code often don’t know which log level to use. Have a document ready which explains which log level developers should use. For example SEVERE should only be used for technical problems which need immediate action. ERROR should be used for errors that someone needs to look into and fix, like not getting a databasde connection, low resources or failing integration points. This is specific to your company and application.

4. Do not log locally

If your server has major problems like resource troubles, it’s often impossible to log in. Therefor you can’t get to your logs finding the problem. Logs should be written to a network drive, copied over to another host or written to the network e.g. with Syslogd. A nice solution is to use the Spread Toolkit to write to a network group with multicasting. This also enables easy monitoring (see “Scalable Internet Architectures”).

5. Monitor your logs

Similar to “Look into you logs”, you should setup a monitoring solution which looks at SEVERE entries, ERROR entries, exceptions and other conditions in you logs. With Spread it’s easy to add monitors. A good idea is also to classify and count exceptions, then do something about the severe and most frequent ones.

6. Use a human readable format

Developers often don’t think about the output they produce. This leads to hard to read log files. “Release It!” has an example for readable output:

[8/14/06 8:22:14:653 CDT] 0000a SSLComponent I CWPKI00001I: SSL service not available
[8/14/06 8:22:14:813 CDT] 0000a WSKeyStore   W CWPKI0041W: One or more key

This row oriented format makes it easier to fast scan logs. Compare this to the your logs.

7. Use error codes in logging

Each cause which leads to log output should have a unique error code. Without a unique error code it’s hard to find the cause in your source code. Error codes make it also much easier to count and classify log statements and enables communications between development and operations.

Want to know more? Books with good sections on web site logging are “Release It!” by Michael T. Nygard (really excellent book!) and “Scalable Internet Architectures” by Theo Schlossnagle.

You can leave a Reply here. Of course, you should follow me on twitter here.

You can share this post!
Do you want to tell others about this article? Use the social bookmark icons to submit this artice to the service of your choice. Thanks.

About the author: Stephan Schmidt is head of development at brands4friends. He has more than 15 years of internet technology experience and 10 years experience in agile. He was head of development, consultant and CTO and is a speaker, author and blog writer. He specializes in organizing and optimizing software development helping companies by increasing productivity with lean software development and agile methodologies. Want to know more? All views are only his own.
Leave a reply.

Comments

Mladen Miljic

Hi Stephan,
thanks for the hints. Can you say something about client side loging? How to log Exceptions occured in the browser (for example) and bind them to a corespending Exception on the server? (Ajax, DOM manipulating, Reporting…)

stephan

I’d say it’s very important to always log a session/user id to find corresponding entries in different logs (especially when crossing boundaries client< ->server, tomcat< ->RMI backend etc.)

Reporting: Call a backend with REST and log the event

If you want to log client exceptions (and reporting), it’s best to not log every event indivudually but collect events and say post 10 events to the backend. Most often it’s not that important when losing frontend exceptions compared to server load.

Michael Campbell

I think a refinement of #1 is, “In a production environment, ensure a way to turn logging on and off.” When debugging a production issue, having debug and/or trace level logs can be invaluable.

Also, much is made of debug logs “slowing down apps”. I’m sure it does, sometimes, in some situations, but this is something that needs to be benchmarked. I’ve seen apps that produce a few hundred megs of debug logs per second that when turned down… had no statistically significant performance gain. This is particularly true with widely distributed infrastructures. If everything is on one box, the story could be very different. The point is, there is no “best practice” here in terms of performance. You gotta test it.

stephan

@Michael: What I’ve learned is that you should be able to turn everything on or off in production, this includes logging.

I find it more useful to have lots of monitoring points in production to find problems than debug statements with logging.

” I’m sure it does, sometimes, in some situations, but this is something that needs to be benchmarked.”

Surely it depends on the debug logging culture (how many, what, who expensive are the logged expressions) in a company on how much it slows down your application. But I’ve seen 30%-50% performance decrease due to turned on debugging.

“You gotta test it.”

Yes, as always. But as a rule of thumb, better have it turned off.

Dimitris Andreou

(a unique, a university, not an unique, or an university. sorry for the pedantry)

stephan

@ Dimitris: Thanks, appreciated.

Make the logs machine readable. Programs a much better at looking at massive amounts of data than I am.

stephan

@Kirk: What does machine readable mean? (Beside one-liners for easier grepping and awking)

But if you do not know what you are looking for, machine readable doesn’t help, you need human readable.

@Kirk: What does machine readable mean? (Beside one-liners for easier grepping and awking)

it means create logs that are easily parsed, greped or awked.

But if you do not know what you are looking for, machine readable doesn’t help, you need human readable.

right, but I often know what I’m not looking for ;-)

Banjollity

Google “splunk” – an excellent log management tool.

stephan

Splunk is excellent – a little pricy for smaller companies though.

[...] Schmidt writes about logging in his latest article. I think he has 7 mostly valid points and even 7 more in an older post, but there is one advice [...]

Hi,

your articles about logging are very good. I wished I had written them :)

I have written an article with my personal comments about your article:

http://myossdevblog.blogspot.com/2009/03/how-to-do-logging.html

Best Regards
Johan

french_c

Seems I have to add another important rule: Excluding stack traces never ever use more than one line to log. In other words: Log statements may not contain an CR/LF – especially in production.

I personally don’t like the error code approach. While it sounds reasonable it becomes tricky in large scale applications. And what do you do with all our 3rd party libraries that don’t follow this rule?

french_c

Should have read the comments though;)

“(Beside one-liners for easier grepping and awking)”

Its very nice and informative article.
@Michael It does decrease the performance after enabling debug in production in most of the cases even after load balancing.
But as debug is very much required to trace the errors, we can have a replica of prod deployed some where as pre-prod which points to same database and is debug enabled where we can leave the prod server to be just error enabled. so the developers can easily replicate the problems there on pre-prod without compromising the performance of production server.

Hey Stephan,

I was reviewing the Spread toolkit and am curious as to how you think it would be used for something other than application-to-application messaging. When I saw the link, I thought, “Wow, there might be an alternative to Syslog that supports larger messages and could be a log event retainer!”

Thoughts?

stephan

In the book mentioned, Spread is used to log all messages to, and then add listeners which will either write to file or monitor special events. Those are easier to write than log analyzers that issue actions (so well it is kind of application-to-application messageing).

Looking at the Second Life Message Queue Evaluation

http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

they use RabbitMQ for this:
“RabbitMQ is already installed in our image and is currently being deployed by the data warehousing team (Ivan in particular) to handle syslog message delivery.”

Leave a Reply

What people wrote somewhere else:

Additional comments powered by BackType

Guide to CodeMonkeyism

Over the last 4 years I wrote many articles on this blog. To make it easier for you to find the relevant ones, I've organized them into topics.

Top 10

6 reasons why my VC funded startup did fail

Go Ahead: Next Generation Java Programming Style

Java Interview questions: Write a String Reverser

The dark side of NoSQL

7 Bad Signs not to Work for a Software Company or Startup

Is Java dead?

Scala vs. Clojure

Never, never, never use String in Java

No future for functional programming in 2008 – Scala, F# and Nu

Clojure vs Scala, Part 2

Job Seeker

Another Good (Java) Interview Question

7 Bad Signs not to Work for a Software Company or Startup

Java Interview questions: Write a String Reverser (and use Recursion!)

Java Interview questions: Multiple Inheritance

As a Manager: What I value in developers

Top 10 Tips (+1) to Get a Pay Raise

Java Developer

Is Java Dead?

Go Ahead: Next Generation Java Programming Style

Be careful with magical code

All variables in Java must be final

Never, never, never use String in Java

Bending Java: More readable code with methods that do nothing?

Startup/CTO

Development Dream Teams

6 reasons why my VC funded startup did fail

American vs. European style of Software Development

12 Things to Reduce Your Lead Time and Time to Market

The high cost of overhead when working in parallel

Essential storage tradeoff: Simple Reads vs. Simple Writes

Agilist

What Developers Need to Know About Agile

5 Practices Better to Change in Your Scrum Implementation

Scrum is not about engineering practices

ScrumMaster and ZenMaster: The joke of certification

What is Trans-Scrum?