Why is it important to move from craft to engineering?

Doing so will help us cope with the ever-increasing challenges of a more automated, more interconnected world—where small improvements in software performance can make the difference between profit and loss; where a reputation for robustness, scalability, and security can add millions to the share price; and where software is more and more the public face of the business.

The codified knowledge and professionalism of an engineering discipline are necessary for:

˲Sustaining and growing delivery capability through changes in technologies, teams, and suppliers.

˲Predictably scaling operations from early prototypes to global rollouts.

˲ Taking control of investments and knowing when to pivot to solutions more likely to deliver favorable returns.

˲ Systematically growing the levels of reuse and interoperability of solution components and systems.

˲Producing long-lived solutions with affordable costs of ownership.

Perhaps not everybody needs to move from craft to engineering. As Mary Shaw says, “The greatest need for engineering discipline exists for software systems that are fully automated and are operating unattended and where the consequences of failure are catastrophic. Examples are telecom equipment, nuclear safety devices, medical implants, self-driving cars, and stock-trading programs. The need for engineering in software development depends upon how serious the consequences are when things go wrong and whether human beings can take action in time to minimize the consequences.” There is also a strong need for engineering systems used for e-commerce, finance, electronic medical records, and even human resources. The consequences of failures in such systems may not include the immediate loss of life, but they can still be “catastrophic” to either the businesses or the individuals affected.

Thus, for many organizations and software systems craft is not enough.

The good news is there is a way for-
ward that maintains the values of agile
while making software development
more of an engineering discipline than
a craft. It involves:
their teams change and their systems
mature.

Another frequent example of unsus-tainability is in the way many companies are facing an uncontrolled explosion in the number of applications they have to support and the overall cost of ownership of IT as a whole.

So industrial-scale agile requires much more than just being able to scale agile. It also means taking a disciplined approach to ensuring IT investments result in sustainable benefits for both the producing organization and its customers.

This involves adopting a different approach to many aspects of agility. We need to look beyond small-scale agile, beyond independent competitive islands of agile excellence, beyond individual craftsmanship and heroic teams, and beyond the short-term instant gratification that seems to be the focus of many well-intentioned but self-centered agile teams. It is this adoption of a more holistic approach that we call moving from craft to engineering. (See Jacobson3 for more background.)

From Craft to Engineering

The move toward agility has led to many benefits for the software industry. It has broken the tyranny of the prescriptive waterfall approach to software engineering, an approach that was causing more and more large project failures, and it has allowed software developers to keep up with the ever-increasing demand for more innovative IT solutions.

It has enabled many companies to do great things but in many cases has led to a culture of entitlement, heroic programming, and short-term thinking that threatens the sustainability of the parent companies and the IT solutions on which they depend. Little or no thought is put into maintainabil-ity, the heroes become potential single points of failure, and the cost of keeping the lights on just keeps growing and growing.

What is needed is a way to maintain the values of agility while making software development more an engineering discipline than a craft—a new form of agile software engineering fit for the Internet age.

What are craft and engineering?

The term craft is usually applied to peo-
ple occupied in small-scale production
of bespoke goods and trades where
skills are passed in person from master
to apprentice. Engineering, on the other
hand, is defined by Wikipedia as “the
application of mathematics, empiri-
cal evidence and scientific, economic,
social, and practical knowledge in or-
der to invent, innovate, design, build,
maintain, research, and improve struc-
tures, machines, tools, systems, com-
ponents, materials, and processes.”
There have been many discussions
about whether or not the term engi-
neering should be applied to software
development and whether or not soft-
ware engineers are actually engineers.
With the rise of cloud computing, big
data, and the Internet of Things, how-
ever, it is clear there are many types of
software and many aspects of software
development that would benefit from
an engineering approach.
In her 1990 seminal paper, “Pros-
pects for an Engineering Discipline of
Software,” 7 Mary Shaw suggested a defi-
nition of software engineering would
include these clauses: “Creating cost-
effective solutions … to practical prob-
lems … by applying scientific knowledge
… building things … in the service of
mankind.” She also said about software
work that “most tasks are routine and
not innovative,” but it “is treated more
often as original than routine,” imply-
ing that there is a lot of potential for im-
proving quality and shortening time to
market “if we captured and organized
what we already know” by codifying our
knowledge, possibly even automating it.
Her observations are still highly rele-
vant; at the Go To Amsterdam 2015 con-
ference on software development, she
talked about the progress made toward
establishing a software engineering dis-
cipline. According to Shaw, the charac-
teristics of engineering are as follows:
˲ Limited time, knowledge, and re-
sources force decisions on trade-offs.

˲ The best-codified knowledge, preferentially science, shapes design decisions.

˲ Reference materials make knowledge and experience available.

˲ Analysis of design predicts properties of implementation.

Although software development shares many of the characteristics of an engineering discipline, we are not there yet. The rise of agile is not a problem unless this is where we stop.