Debugger

From APIDesign

Current revision

Debugger is a program that helps to find out what other programs are doing.

Motto

"Every sufficiently advanced technology is indistinguishable from magic" ... until you debug it.

Plot

Often people use debugger to understand why their own code does not do, what they would like it to do. However debugger is incredibly useful for other purposes as well. Here is its story.

Building Modular systems

Chapter 1 of TheAPIBook discusses the art of building modern software systems and concludes that these days we are building systems that we don't understand. As an example it mentions that in order to write a dynamically generated web page one needs to understand HTML and possibly a Servlet API, but you don't need to have knowledge of anything that lays below: Glassfish server, Java VM, Unix libraries, Linux kernel, the VirtualBox nor even the hardware. At the end you can deploy a massive solution and in fact understand roughly one percent of it!

Some people may find it freightening that we use systems without real understanding, but The Art of Building Modern Software chapter concludes that such level of cluelessness is OK and in fact completely necessary. Brain has limited capacity, and the learning time is finite as well. As such we just can't understand everything. It is enough to know just the surface:

• the Servlet API - but not its actual implementation in the Glassfish server
• the user or command line interface of VirtualBox - to install Linux into it and to launch it
• the apt CLI - to install Java and Glassfish

Knowing these interfaces is enough. The details behind them are unimportant (if things go well).

Btw. this is also the reason why API design is important (command line parameters, as well as layout of files, are as important types of API as classes and their methods). Well designed application programming interfaces encourage cluelessness. The less knowledge people need to have to use a technology, the better the technology is. That is why it is still useful to read my Practical Design Book and learn how to design APIs properly! But let's now get back to topic of debugging.

Cluelessness of an API User

I am using word cluelessness for the blessed state when one does not need to know the internals behind APIs. However in fact I mean selective cluelessness: when there is no need to know to make things work, the less we know the better.

However when things go wrong, then we should concentrate on what is wrong and get as deep knowledge of the affected area as possible. We should select that we want to know everything about a particular part of (for example) Glassfish and learn every detail about it (for example why it throws some NullPointerException).

These days it should not be hard to increase our knowledge about our libraries. Most of them are open source and thus getting access to their source code is easy. Putting a breakpoint on appropriate line, launching own application under debugger is usually easy as well. Moreover it is also easy to play with the code - modify a line, compile (using Maven or make) and use the modified version. There are no barriers to increase our knowledge about Glassfish, Java, Linux, etc.

In spite of the above easiness, I don't see people doing it. I don't see them looking at sources of libraries they use. Rather than that they treat them as blackbox. In spite of all the options we have, some of us decided to stay clueless (in the negative meaning, without the important selective part).

The Googling Robot

Rather than trying to investigate the problem, the first reaction is to query google search machine. I have to admit, I do it too and the results from Stackoverflow web site and really valuable. It is selectively clueless to check whether somebody else tried to solve similar problem, so far this attitude is good.

However the really problematic state occurs as soon as the query yields no results. I've seen many situations when desperate programmers frustrated by unsatisfying search gave up! Such googling only robots will however face inevitable destiny. These days google can recognize voice, drive a car - everything done by a brute force searches for correlations. The day when google will be able to create artificial intelligence that will program using searches is not far away. At that moment programmers will either choose to go out of their previous business or will have to become selectively clueless again - and learn how to use debugger.

Btw. you may object that one day an AI that knows how to debug will be created as well. I am sure somebody will try, but first of all such day is further away. Second, let us remind the foundation of computer science - the Halting problem - which shows that there can't be a program which would understand behavior of all programs - e.g. at least some need for human programmers will always remain.

Knowing Every Detail

Once a customer of the NetBeans Platform invited me for a consultation. It is not common to send the creator of a framework for a consultation, the usual strategy is to send the junior guy to find out if they can spell Java. Later send somebody more experienced and only if every effort fails, ask the guru. But even for me, as a creator of the NetBeans Platform, it is important to see real customer from time to time, so I took the opportunity to visit such an important customer.

Certainly I can talk about architecture decisions and explain how to use NetBeans APIs on a higher level, but when it comes to details, I may often be in total cluelessness situation (there is a lot of NetBeans APIs check at http://bits.netbeans.org/7.3/javadoc/ and we design them as a team). As a result my most common answer to customer questions was: I don't know. Let's put a breakpoint somewhere!

Most of the time of my visit was spent debugging. We located misbehaving API source code, placed a breakpoint somewhere and run the whole application. After few steps over we usually know what is the problem and could either workaround it, or (as I am not afraid to enhance API compatibly) modify the NetBeans source code to better suite customer needs. At the end the customer was satisfied - but was it really necessary for me to be there? Could not the guys use the debugger themselves?

When my visit was over I ironically commented that I did nothing else than debugging and patching NetBeans. At least I was useful to have the rights to make sure the patches get included in next version, I concluded. The customer's reply was shocking: No, you are more valuable for knowing where to place a breakpoint!.

Does the ability to place a breakpoint becomes the most important skill for a modern developer?

When Things Don't Happen

Debuggers were designed to allow developers to step through their own code. Certainly, when such feature hit market for the first time, it was a huge step forward. Ability to understand the behavior of own program by stepping through line by line and seeing the state of memory and variables is certainly useful. Much easier than running the whole program and then trying to guess on which line something went wrong.

However the proliferation of open source libraries and the fact that these days we rather assemble applications than write them makes us use debugger on foreign code rather than our own. Moreover we are no longer using just simple libraries (where we just made calls to them and continued our execution when the call returned), we are using frameworks. As a result it is the framework that makes calls to us depending on some declarative registration. Debugging own code is still the same, however when one needs to find out why something did not happen - why the framework decided not to call us - we may get into tougher situation. Are our debuggers good enough to help us solve such problems?

Not really. Debuggers were originally designed to help us fix our own code when it misbehaved. Using them to find out why things don't happen is not easy.

Side by Side Debugging

Few years ago I was working on a project to bring JDeveloper and NetBeans closer to each other. We started by replacing JDeveloper's module system based on Equinox by Netbinox (a mixture of Equinox and NetBeans that was the fastest OSGi container on the planet at that time). You can imagine replacing something as fundamental as base module system involves a lot of debugging.

The best approach was to run both systems side by side and watch for the differences. For a certain period of time we had a possibility to run the JDeveloper either on top of Equinox or on top of Netbinox - just by passing in different system property. When something went wrong under Netbinox, I performed the same operation in Equinox, observed the behavior by stepping through the affected method line by line and then compared the behavior with Netbinox. Usually it was quite obvious where the difference is and yes, there were few.

This kind of side by side debugging is useful for finding out what is wrong when things don't happen. Because we had the old Equinox version which supposedly worked fine one could put a breakpoint to the place that should be executed, when the breakpoint is hit in the old version, just remember the stack. The problem in the new Netbinox version had to be along the stack. Placing another breakpoint (or breakpoints - depends on luck) to methods higher on the stack must once reveal a place where the code behaves the same in both versions. Then one could again switch to line by line debugging and find out where the difference between Equinox and Netbinox is.

Side by side debugging is not useful only when we are rewriting one system to a new version. Even if something is wrong when we are using a framework, we can usually find a tutorial or demo application that is supposed to perform similar thing as our application. At that moment, we can compare the behavior of working demo, with misbehavior of our application. Side by side debugging is useful for everybody.

Insufficient Documentation of Maven Tasks

Do you know the difference between Ant and Maven? In Ant you describe what to do and make calls to individual Ant tasks. On the other hand in Maven you describe what should be done and let the system do it for you. That is more declarative, but also more framework like: often it is hard to find out why some actions are not performed the way you'd like them to be.

Recently we were writing a Maven plugin in Ant (yes, that is possible, both projects are supported by Apache foundation and as such they seek some synergy). Documentation how to achieve this is available, however it covers only the most common usecases. When you need to take it to extreme, use not only simple Ant script, but rather a set of files that work in harmony, you need to understand what is happening. The only way to do it is to use debugger. Can you debug your build?

The good thing about Maven and its NetBeans IDE integration is that debugging your build is easier than ever. Just execute your build (by invoking any goal on your project), let it fail and then restart it in debug mode. Press the yellow arrow in the output window and when a dialog pops up, select debug maven build. See the picture for visual explanation.

Knowing how to debug your system is an essential knowledge what allows you to worship selective cluelessness. This applies not only to Maven, but to any system one happens to work with.

Debugging without Source Code

Debugger can be useful even for understanding a system without a source code. It is just essential to know where to place a breakpoint. Once I was trying to understand the behavior of iOS's WebUI - of course its not open source, but luckily Gdb (as well as many other debuggers) can place a breakpoint to a method and stop the execution when the method is called. As we needed to find out why the WebUI does not load a resource using our URL, we added a breakpoint to constructor of NSURL. As soon as the breakpoint was hit, we could inspect the stack, see names of methods and even values of registers. Based on that we could get a clue of what was going on.

Once I was inspecting a closed source Java implementation of a Mylyn connector API. The system had three layers: User interface and connector API was open source, the library in middle was not. By placing breakpoints into the bottom connector API I obtained very good understanding of what the blackbox in middle does. It is all about knowing where to place the breakpoints.

Have your Java application exited without a reason? Try to place breakpoint to System.exit(). If that does not work, create an exception breakpoint to stop when an exception is thrown.

Trying to influence behavior of your application with a system property and it does not work? Place a conditional breakpoint to Properties.getProperty(name) to stop when name is equal to name of your property.

Curious to know why a communication to a HTTP server fails? Place a breakpoint to URL constructor. Etc.

Use the Debugger. Stupid!

I am glad when people use NetBeans Platform, however it drives me mad when they approach me with a question that could easily be answered by debugging. In such situation I am tempted to say: Use the debugger stupid!. Usually I choose different words, but in case you see me to close a bugzilla issue, or reply to an email with a reference to debugger, you know what you should do: Use the debugger, stupid!