Communications - April 2009

The “.” variable evaluates to the input address used when applying the macro, and the “+” variable evaluates to that input address incremented by the byte count of all preceding format characters. The macros were then maintained with the kernel source code.

More than a decade later, in 1997, I was working at Sun on what would become Solaris 7. This release was our first 64-bit kernel, but the kernel-debugging tool of choice was still adb just as it was in 1984, and our source base now contained hundreds of useful macro files. Unfortunately, the implementation of adb was essentially impossible to port cleanly from 32-bit to 64-bit to debug the new kernel, so it seemed the time was ripe for the development of a new clean code base with many more modern debugger features.

As I considered how best to approach the problem, I was struck by the fact that despite its brittle, unstructured code base, the key feature of adb was that its syntax was imbued deeply in the minds and behaviors of all of our most experienced and effective engineers. (As someone aptly put it at the time: “It’s in the fingers.”) So I set out to build a new modular debugger (mdb) that would support an API for advanced kernel debugging and other modern features, yet would remain precisely backward-compatible with existing syntax and macros. Sophisticated new features were added after a new prefix (“::”) so they would not break the existing syntax (for example, “::findleaks” to check for kernel memory leaks). The entire syntax was at last properly encoded as a yacc parser. Macro files were phased out in favor of compiler-generated debug information, but the “$<” syntax was left as an alias. Another decade later, mdb remains the standard tool for postmortem debugging of the OpenSolaris kernel and has been extended by hundreds of programmers.

The debugger tale illustrates that a little purpose-built language can evolve essentially at random, have no clear design, no consistent grammar or parser, and no name, and yet endure and grow in shipping operating systems for more than 40 years. In the same time period, many mainstream

figure 5: Debugging the “proc” structure on SunoS 4, circa 1984.

address $<proc

./”link”16t”rlink”16t”nxt”16t”prev”nXXXX +/”as”16t”segu”16t”stack”16t”uarea”nXXXX +/”upri”

+/”upri”8t”pri”8t”cpu”8t”stat”8t”time”8t”nice”nbbbbbb +/”slp”8t”cursig”16t”sig”bbX +/”mask”16t”ignore”16t”catch”nXXX +/”flag”16t”uid”8t”suid”8t”pgrp”nXddd +/”pid”8t”ppid”8t”xstat”8t”ticks”nddxd +/”cred”16t”ru”16t”tsize”nXXX +/”dsize”16t”ssize”16t”rssize”nXXX +/”maxrss”16t”swrss”16t”wchan”nXXX +/16+”%cpu”16t”pptr”16t”tptr”nXXX +/”real itimer”n4D +/”idhash”16t”swlocks”ndd +/”aio forw”16t”aio back”8t”aio count”8t”threadcnt”nXXXX

figure 6: fortran and Ratfor, circa 1975.

if (a .gt. b) goto 10

sw = 0 write( 6, 1) a, b goto 20

10 sw = 1 write( 6, 1) b, a

20 …

if (a <= b) { sw = 0;

write( 6, 1) a, b } else { sw = 1;

write( 6, 1) b, a

}

figure 7: c and Algol mutant from early “adb”.

#define IF if(
#define THEN ){
#define ELSE } else {
#define FI }
#define ANDF &&
...
localsym(cframe)
L _ INT cframe;
{

INT symflg;

WHILE nextsym() ANDF localok ANDF symbol.symc[0]!=’~’ ANDF ( symflg=symbol.symf)!=037

DO IF symflg>= 2 ANDF symflg<= 4
THEN localval=symbol.symv;
return(TRUE);
ELIF symflg== 1
THEN localval=leng(shorten(cframe) +symbol.symv);
return(TRUE);
ELIF symflg== 20 ANDF lastframe
THEN localval=leng( lastframe+2*symbol.symv- 10);
return(TRUE);

FI

OD return(FALSE);

}

languages came and went into the great beyond (Algol, Ada, Pascal, Cobol, and so on). Fundamentally, this debugger has survived for one reason: it concisely encoded the exact task its

users performed and thereby connected to those users. Take an address, dump out its content, find the next address, follow it to the next location of interest, dump out its content, and