7 More Good Tips on Logging

Logging in web applications is important - to know what's going on, for performance tuning and incident analyis. This is my second post about logging. The first post "7 Good Rules to Log Exceptions" was specific to logging exceptions, ths is about logging in general. What makes your logs more useful to you?

Nerdy Bookshelf Part One
Creative Commons Licensecredit: schoschie

1. No debugging logs in production

I have seen time and again that debug logging is enabled in production. This can be intentional or happening by some developers who accidently checked in a debugging logging configuration. Enabled debugging slows down your application remarkedly and makes it impossible to read production logs due to noise. Make sure during deployments - best with some scripts - that debugging level logging is disabled during production.

2. Look through your logs

Some companies have good logging in their production system, but do not look into their logs. Look into your logs, discover issues (bugs, performance, memory) with your application and fix them. Essentially your logs should be without known errors.

3. Log to the correct log level

Developers who write logging code often don't know which log level to use. Have a document ready which explains which log level developers should use. For example SEVERE should only be used for technical problems which need immediate action. ERROR should be used for errors that someone needs to look into and fix, like not getting a databasde connection, low resources or failing integration points. This is specific to your company and application.

4. Do not log locally

If your server has major problems like resource troubles, it's often impossible to log in. Therefor you can't get to your logs finding the problem. Logs should be written to a network drive, copied over to another host or written to the network e.g. with Syslogd. A nice solution is to use the Spread Toolkit to write to a network group with multicasting. This also enables easy monitoring (see "Scalable Internet Architectures").

5. Monitor your logs

Similar to "Look into you logs", you should setup a monitoring solution which looks at SEVERE entries, ERROR entries, exceptions and other conditions in you logs. With Spread it's easy to add monitors. A good idea is also to classify and count exceptions, then do something about the severe and most frequent ones.

6. Use a human readable format

Developers often don't think about the output they produce. This leads to hard to read log files. "Release It!" has an example for readable output:

[8/14/06 8:22:14:653 CDT] 0000a SSLComponent I CWPKI00001I: SSL service not available
[8/14/06 8:22:14:813 CDT] 0000a WSKeyStore   W CWPKI0041W: One or more key

This row oriented format makes it easier to fast scan logs. Compare this to the your logs.

7. Use error codes in logging

Each cause which leads to log output should have a unique error code. Without a unique error code it's hard to find the cause in your source code. Error codes make it also much easier to count and classify log statements and enables communications between development and operations.

Want to know more? Books with good sections on web site logging are "Release It!" by Michael T. Nygard (really excellent book!) and "Scalable Internet Architectures" by Theo Schlossnagle.

Scrum is not about engineering practices

Scrum is not about engineering practices - which is a good thing. Martin Fowler writes:

They adopt the Scrum practices, and maybe even the principles
After a while progress is slow because the code base is a mess

and connects Scrum failure to missing engineering practices. This completely misses the point. Scrum is not about engineering practices, it's about management.

Engineering practices are a responsibility of the team. Scrum creates self organized teams. They organize themselves, they organize their quality. They organize their tools. They organize their engineering practices (TDD, pair programming). Why? Because they are responsible for delivering. No one else is.

Craftsmanship and level of done

Often quality and craftsmanship are organized by a level of done agreement in the team which describes what a team considers when code is done. This can include

  • Documentation / JavaDoc
  • Functional tests
  • Unit tests
  • Bug free
  • Refactored, maintainable code
  • Reviewed code

What about technical debt? Technical debt is not professional. Seeing an ice berg and keeping a crash course is not professional. With the right level of done there will be no technical debt.

Scrum helps with good engineering practices

Concerning Marting Fowlers comments Dean Wampler twitters:

His comments match my experience at client sites. Teams using Scrum w/out the XP prog. practices don't work long-term.

Not very insightful it says: developers who do not use professional practices will fail. Of course they fail, but they fail independently of the process you use.

The main difference to classic project management is that developers have the freedom to define their level of done and the amount of work they do. Developers define what stories they work on in each sprint, management doesn't set a (unrealistic) finishing date as often happens in classical software projects.

Is there too much quality? Doesn't the product owner care if the team takes "too much" time for quality? The product owner is entitled to two things:

  • Story estimates
  • Shippable products
  • Professional developers

The product owner is not entitled to speed. Scrum sets resources and time in the Iron Triangle and let's developers decide about scope. Speed is scope / time and is an output variable of the team, not an input variable. If she thinks the velocity of the team is too low, talk to the ScrumMaster and remove impediments. But engineering practices should never be dropped. Other crafts will never drop theirs. Ask a doctor to drop sterilizing to gain speed. Ask a banker to drop double-entry bookkeeping. They won't. Neither should developers drop theirs [1].

[1] This could mean developers need to be trained to be professionals. But a company needs to teach professionalism to developers independently of what process it uses. If you do not have skilled, professional developers you're doomed and no process will help you.