Communications - July 2010

went through the code line by line and
took some notes. Unfortunately, these
notes are not in electronic format, and
worse…they’re in Chinese.”
There may have been a specifica-
tion originally, but the source code
evolved over time, while the spec was
never updated. To mitigate the diver-
gence of spec and source code, we
looked at making source code, even
Fortran, as readable as possible and
interleaving source code with specifi-
cation or documentation. We tried an
implementation of the HPCS graph
analysis benchmark, SSCA #2, where
the “source code” was HTML from
which a script could extract Fortran
code to compile and execute. This ap-
proach to having a single artifact to
maintain, instead of disjointed speci-
fication and source code, is similar
to ideas found in Mathematica note-
books, Donald Knuth’s Literate Pro-
gramming, and Scientific WorkPlace.

Validation is also difficult. One must compare results in particular parameter regimes to results that might be known from analysis or predecessor codes. Since validation is so expensive and depends so critically on experienced scientific understanding and intuition, over most of an HPC application’s lifetime one simply checks software modifications by comparing results with an earlier version of the code. Whereas the science is meaningful to only limited precision (say,

1% or even 10%), checking numerical
results in HPC usually means check-
ing fickle floating-point arithmetic
out to the least significant digit. We
found cases, for example, where we
refrained from changing the source
code because changing (( 2*pi)*k)/N
to 2*((pi*k)/N) or changing X*(1/
deltat) to X/deltat changed float-
ing-point results subtly. We do not
know if the results were more accurate
or less, only that they were slightly dif-
ferent. These differences prevented
us from making the source code more
readable or run faster.
gramming habits develop in a culture
of fast prototyping, avoiding advanced
language features since their support
is immature and focusing instead on
the last drop of performance. As pre-
sented in Donn Seeley’s ACM Queue
article, “How Not to Write Fortran in
Any Language” (December/January
2004/2005), examples of poor pro-
gramming practices abound.

Programming for verifiability is often not a priority, as the BT example illustrated.

As another example, in sPPM we found thousands of lines of code for handling boundary conditions. The rewritten code used only about a dozen lines. There are many reasons for this astounding reduction, but one issue is that the original code attempted to fill in “ghost cells” only when their values would be needed. (Ghost cells are replicas of real computational cells, where such replication can simplify the handling of boundary conditions.) In the rewrite, we would routinely fill in all ghosts cells. Eliminating checks on whether such updates were needed facilitated the programming logic immensely, with nearly no overall performance loss in the cases we studied. In HPC, the mind-set is usually to program for performance rather than programmability even before establishing whether a particular section of code is performance sensitive or not.

The ISO/IEC standard on software maintenance adopted the term perfective maintenance. Modifying source code simply to improve its maintainability, however, often receives scant attention when other objectives— such as fixing defects, implementing new features, tuning performance, and migrating to new platforms— clamor for attention.

The NPB BT source code takes
hundreds of lines of code to compute
the time derivative dU/dt to form the
right-hand side. This computation ap-
pears to have been implemented from
scratch twice, once in file rhs.f and
again in exact _ rhs.f. Even if this
duplication of effort was overlooked
originally, perfective maintenance
should weed out such redundancy to
benefit the generations of HPC work-
ers who have had to look at this source
code since it was first written—provid-
ed, of course, that this is important for
the software’s owners.

Where Do We Go from here?

Repeating some of these programmability studies on larger HPC programs would be interesting. In particular, it would be nice to move from self-contained programs that are small enough for one person to have written—what DeRemor and Kron would term “programming in the small”— to larger pieces of software, written by many people and where interfaces among many parts are important (“programming in the large”). Like nature, source code looks different at different scales: from fast prototyping, to self-contained applications, to multi-decade legacy code. Further work to relate source-code characteristics empirically to human productivity metrics would also be interesting.

Most of all, the HPC community on all fronts: language development, compiler maturity, hardware innovations, HPC software development practices, and even procurements and competitive benchmarking.

When we start with an existing language, however, we benefit from available compilers, systems, reference codes, experience, and programmers.