How to design and build software
Might as well have a pretentious title, yes? After
experience with both successes and failures, I do have some ideas
about the factors in making it work, but how I got these ideas
was not at all mainstream.
I spent the period from August 1999 until January 2001 in a state of semi-continuous
international business travel, often with our company's Vendor Relations Manager. Long flights
led to boredom, and it was this experience that led to my taking up crossword puzzles
for entertainment. Pen. Paper. Time.
Despite my lower skill level, it did not take long to confirm that
together we could work the puzzles in less than half the time that
it took either of us to work a puzzle alone.
This realization proved quite a contrast to what might be expected
on the part of the majority of people in the field of IT. Indeed, we
are used to seeing the tremendous non-linearity of adding people to a
late project only to make it later.
Fred Brooks first pointed out this
paradox to us in
The Mythical Man Month over thirty years ago.
What makes a crossword puzzle partitionable, and can we thus design
software differently than we do? Note that this question is unrelated to
the usual, and sometimes far-flung analogies that we see about how
software development is like some other activity, often building
a house or manufacturing some large material product. Instead,
let's admit that software development is what it is, and see what we
can apply from learning about other types of activities.
Properties of the Crossword Puzzle
Here are some relevant facts about
cruciverbalists working as a team:
- All the requirements were known. Certainly it is possible that
you might object to this first characterization, and be working on a
diagram-less crossword and thus not have all the requirements. But,
the common garden variety crossword has the layout of the squares,
the clues, the equivalent of an entity relationship diagram that maps
the clues to the answers, and tells you where each goes. This is quite
a bit more than almost any software project has when it begins.
- Each solver has a “perfect” and identical copy of the requirements. While
we were working on the puzzle, we each had the same copy of the
requirements continuously available for our use. Additionally, the
requirements did not change while we were working the puzzle.
- Learning from previous puzzles is directly applicable to the work
at hand. Anyone who has worked crossword puzzles can relate to the
degree to which experience improves skill level . . . rapidly. As
one progresses, one is able to solve new ones and harder ones in
less time.
- The puzzle need not be worked serially. In other words, there
is no need to start with in the upper left hand corner with 1-Across
and 1-Down and work through the clues in order. In fact, the best way
to tackle a crossword puzzle is to do something that seems natural:
find some region where you know the answers to several clues, and
work your way out from that starting point.
- Solving any part of the puzzle helps to find the solution to at
least one other part of the puzzle. Very often, the best progress
can be made by jumping around within the puzzle, and then returning
to work on harder entries once a few letters are discovered.
Oswald Jacoby,
a master of everything from cryptography to crossword
puzzles and duplicate bridge, referred to this natural tendency as
“bold guessing.”
- The fact that our background knowledge differs is an asset. In
our case, one of us has an educational background in organizational
psychology, and the other in software science. We are male and female;
our ages differ by eighteen years; we read different magazines; and
we grew up on opposite sides of North America. This difference in
background allows us to know more together than we know separately,
particularly when it comes to solving puzzles and confronting other
problems.
- The implementation of the crossword puzzle is lower 'tech' than
its solution. Since its first appearance in the Sunday New York Times
newspaper in 1913, the delivery of crossword puzzles has not changed
much, and the methods have not grown in number.
Even if you prefer to work your crosswords online at
oneacross.com,
the site’s user interface has been modeled on the paper version
that antedates it by almost a century. The solution tools are a pen,
the puzzle book, and the eraser, at least for those of us less than
perfectly skilled in the field of puzzling.
The result of this very simple delivery vehicle is that the solver
may devote himself completely to the solution, rather than spending
many hours mastering the problem’s interface and the solution’s
tools.
- The necessary knowledge and the difficultly of finding it are
correctly divided to support solving the puzzle. In the specific case
of the crossword puzzle, general knowledge is useful, and specific
knowledge is easy to get help with.
Crossword puzzle solving benefits from knowing “
a little bit about a
lot of things” as the lyrics to a well known jazz standard point out
— deipnosophism is another natural trait. If the theme of the puzzle
is a specific subject, such as classical music or sci-fi film stars,
it is relatively easy to find a reference book or just call a friend.
It is much harder to give yourself a crash course in basic history,
geography, the Roman pantheon, Shakespeare, movies, modern sports
figures, and all the other trivia that haunt crossword puzzles. We
might say that crosswords are designed to appeal to the diminishing
number of people who have a classical education.
-
It is fairly straightforward how and where to incorporate prototypes
and mock-ups. You may “pencil in” a candidate answer to the
puzzle, and then see how it works out. If it fits with the interlocking
words, then you can bring out the pen and incorporate the prototype
directly into the final product.
Readers who are appreciative of the terms of art in IT may wish to
note that this is one of the few times when the distinction between
mock-ups and prototypes is not particularly relevant. In fact,
both are easy to incorporate in the final product, whereas both
are frequently dangerous to incorporate in the final products of IT
development efforts.
-
Multiple parts of the puzzle can be worked simultaneously, and one
person can take a rest while the other person continues to work.
That hardly requires comment, does it?
Applying these properties to software development
It is now time to consider how what we know about the relative
ease of teamwork in the world of crosswords may be applied to software
design and development. At the risk of pushing the lesson by analogy
too far, most of the points in the previous section translate quite
directly into a desired property of software development.
- We need to know a fairly complete set of
requirements for any software project to go well.
This comes as no surprise to those of us in the IT business, but we do
not always look outside of IT to determine how other disciplines cope
with what is surely a difficulty common to many lines of work. IT
is frequently myopic, and there is much we could learn from other
disciplines.
Requirements are subject to constant revision while one is developing
software, but many of the problems that result from changing
requirements are directly traceable to the lack of practice that many
developers have with collecting, cataloguing, and presenting the
requirements that they do have.
In particular, the various problems that our discipline has with
requirements are not ameliorated by separating the analysis from the
implementation. Analysts write requirements, but they seldom are the
readers. Programmers read the requirements, but seldom talk to the
customers.
The situation is quite a bit different when one or more people are
working on the crossword puzzle: the translation of the requirements
(right brain) to the solution (left brain) is performed by the
corpus callosum of a brain that is much like the one that created the
puzzle. That translation is immediate, specifically in the sense that
there is no mediator.
- Keeping all requirements continuously available
to everyone is a critical part of software project management and
software design.
The problem with having undocumented requirements is that they
disappear along with the “undocumentor.” The crossword puzzle
case is the equivalent of giving each person on the project a complete
set of requirements even if those requirements are outside of one’s
specialty or part of the project. The assumption is that it helps to
know what everyone else on a project is doing.
Of course, this is not what often goes on in commercial software
projects. Programming is generally a type of work in which one gets
ahead financially by becoming increasingly specialized in one area
only. Programmers regard themselves as increasingly elite as they
begin to work in fewer and fewer areas of the code — generalists
are to be confused with tyros.
As a person who has extended offers of employment to many programmers,
I am struck by the disparity between the vitas that precede the
candidates to the interviews and what the candidates actually seek
to do. The vitas are frequently littered with skills as if the author
thought it best to look in Information Week to
determine what is “hot,” and then to include the list.
As an example, when I have asked about the candidate’s skills with
written English, the candidate’s eyes grow large with shock, and
express almost involuntarily what they really think: requirements
and documentation are job best left to others.
- Choose designs and solutions that build on what
you already know.
It is surprising how often this is overlooked, or perhaps how reluctant
programmers are to march toward delivery instead of stopping to
smell the technological roses along the way. Technology surfing is
something of a tempting and dangerous art. When one must work in
several unfamiliar areas, a tremendous amount of effort is applied
to the accidental difficulties of technology, rather than applying
the experience one already has to the difficulties of the essential
problems.
Certainly, it is true that new technology is being made available all
the time, and that its use is frequently the most expeditious way to
solve various implementation problems. However, building on what
one already knows reduces the complexity in the new design as well
as lowering the risk level.
- Choose designs that allow the
development team work on several parts of the solution at once.
Nothing will please the funding manager or the independent customer
more than seeing bits and pieces emerge as the schedule passes. The
alternatives are seeing nothing as the schedule passes by on the
calendar, or seeing the project delayed while the development
staff must wait for some other part of the software project to
complete.
- Design for testability — Create designs that
allow the completion of one subsystem in advance of the others
to provide some degree of scaffolding for other parts of the
system.
Not all software designers choose to design for testability. This
fact may be a consequence of the increasing "silo-ification" that is
taking place in many shops. If the construction of the software is
itself divided into disjoint groups of people, then surely testing
is the responsibility of someone else.
- Design for analogy.
This is different from designing so that you must have a Noah’s
Ark of programmers from different specialties to complete the
task. Instead, I suggest the choice of a design that is “like”
something in the physical world. By doing so, you can benefit from the
knowledge of non-software activities among the development staff.
One way to think about this idea is that it amounts to a use of
meta-patterns. In the book
Design Patterns, “patterns”
are defined to be the repeated abstractions that are commonly
found in software. Meta-patterns then, are patterns of entire
architectures. Given the amount of time that mankind has been solving
other types of problems, and the short amount of time we have invested
in software construction, our best meta-patterns are doubtlessly
found in disciplines outside of IT. Software systems are, or should
be, manifestations of the meta-patterns that come from other fields
of human activity.
-
Use the simplest possible implementations in your design.
“Keep it simple, stupid,” has always seemed to me an abrasive
phrase. A more useful and pleasant piece of advice came from
Robert Schumann, who suggested that it is more satisfying to hear
a simple piano piece played well, than to hear someone hack at a
composition that is too difficult for the player’s level of skill.
-
Decide on the specifics of the implementation at the last possible minute.
The implementation should, ipso facto, come last. The specific should
follow the general, especially because the construction of any solution
is a process of progressive refinement.
Too often, a project manager or principal programmer starts out with
the implementation in mind. Advertisements for jobs in the software
field give us an excellent clue why this is so. A casual reading
of the ads tells us that many people consider absolute skills with
the various tools of programming to be paramount; why else would the
advertisements be framed in terms of languages and products rather
than ability to choose the languages and products, or better still,
the ability to understand the problem?
Technology has been changing for the last forty years, and there is
no end in sight. The Java programmers will have to be retrenched just
like the CICS, FORTRAN, COBOL, and C programmers before them.
I suggest finding adaptive generalists to make up the core “surgical
team.” Hire only a few specialists, and use contractors for the short
term needs related to the specifics of your solution. Ultimately, it
is cheaper to let the specialists train themselves, and to develop
in one’s organization knowledge of the business, and the ability
to solve problems in that business.
- Avoid serialization — focus on divisibility of
projects and tasks among people.
This is far easier said than done, but if it is not said, it is never
done. It can be quite instructive to look at a Gaant chart print out to
make an approximate survey of how much of the work is serialized. The
battle is between serialization and subsystems, where subsystems are
the desirable item.
What is a subsystem? Try this working hypothesis: a subsystem is
[a] something less than the whole thing, that [b] can be replaced
in the field without replacing the entire system, and [c] from
an architectural standpoint the couplings are tighter inside the
subsystem than between the subsystem and other subsystems. This
working description contains features of several definitions in use
in the IT and other industries.
Learning from this approach
It has been said that analogies are never true or false, they are
simply useful or not useful, probable or improbable. Pressed too far,
all analogies break down.
For example,
Grady Booch has said that it is only at the end of
a project that we understand the requirements. This is an innovative
way of saying that complete requirements will never exist
for a software system the way
that they do for the crossword puzzle.
And there are problems with the method in terms of scale. A crossword
puzzle with two people working on it is not quite the same affair
as a software project with twenty or two hundred people working on it. And in both
the case of the puzzle and the software project, they do not scale
up easily to ever larger problems.
But the crossword puzzle, and the techniques for solving it are
a meta-pattern for one type of successful software design and
construction effort. For years, software engineering has tried to optimize the generally
serial approach to software development, all the while treating it as
a kind of special and unique activity. In some ways it is, but software development is just another
human activity, requiring balance, insight, and judgment.