Safety is not our first priority

I was on a flight back to Australia when something in the routine safety
demonstration video struck me with more significance than it had in the past.

“Safety is our first priority”

I began to realise just how often I had seen this value being espoused and how
often I’ve come to take it for granted. The infrastructure and services I rely
on every day put my safety as their top priority. Above making money. Above
being innovative. Above customer service. If that organisation did not keep me
safe whilst my livelihood was in their hands, the organisation would consider
itself to have failed at its primary objective.

If you look around, you’ll find that this is an incredibly common organisational
value. You will find it in any organisation that operates heavy machinery, any
organisation that is responsible for taking care of a dangerous place and any
organisation that is responsible for building infrastructure upon which the
public relies. It is therefore no surprise that this value has been reflected in
the customary law of negligence for centuries. When someone is vulnerable to
your actions making them unsafe, you are under a duty of care to ensure that
above all else, they are safe.

It is customary for all engineers in the United States and Canada to receive an
iron ring upon their qualification and admission into the profession. It is
folklore that the iron is sourced
from the remains of the Quebec Bridge, which collapsed during construction in
1907, killing seventy five construction workers. The reason the bridge
collapsed was that the engineers responsible for its design and construction cut
corners to get the project to completion more quickly. Those workers were
relying on the engineers to keep them safe and the engineers put profit ahead
of the safety of those people. The ring is intended to be a permanent reminder
to all of those in the profession that such errors should never be repeated.

The priority of safety in software

You will very rarely find this value in organisations that develop software.
Most organisations are even allergic to it, including verbose disclaimers in
software licences purporting to waive any implied warranty that the code you run
is fit for purpose and won’t cause you trouble. Most licenses will require users
to agree to numerous waivers and consent to all sorts of things which may be
harmful to them. There is no way other professions would get away with that
without raising an eyebrow. But for software, we don’t seem to care.

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

This is puzzling, because at the same time, professional bodies representing the
computer software industry purport to put the public interest as their number
one priority. For instance, the Australian Computer Society’s (Australian Computer Society code of ethics) says that "You
will place the interests of the public above those of personal, business or
sectional interests." This disconnect is disturbing when we think about the
consequences that flow from such ignorance. A failure to have a safety oriented
culture means that as so-called "engineers" our profession becomes complacent
about the impact of our strategic and engineering choices upon users. It results
in culture that puts the need to stay ahead and continue making a profit above
the need to protect the legitimate safety interests of users. It leads to a
discourse which implicitly accepts that failure is inevitable and can only be
mitigated as opposed to prevented by making trade-offs which may reduce
competitiveness but protect the public.

A resort to realism

Many computer software professionals would retort at this point delivering
perfect quality software is just not realistic. The project may be on a tight
budget or operating under a tight timeframe. It would not be possible to remain
competitive and maintain such a high standard. And nevertheless, software
professionals say, users have come to understand this reality as well. The
software industry has remained unregulated since its inception and the benefits
for society have been explosive growth of technology and innovation at a pace
never before seen.

Such explosive growth and innovation has had real winners and losers. Over the
last forty years technology has moved beyond the realm of academia and military
research and become democratised. Computing is readily accessible in the age of
the smartphone. It is also no surprise then that the wealth generated by
computer software has become concentrated in the hands of the few who were best
able to execute and facilitate this explosive growth.

But there are losers. It is becoming increasingly difficult, and now almost
impossible for people to opt out of relying on software in their daily lives,
just like it is now almost impossible to opt out of roads, electricity and
plumbing. Software controls an increasingly higher degree of the infrastructure
that we rely on every day and increasingly controls the way that we interact
with each other, do business and access essential services. In an age where
software grows explosively and operates the world, people have become
increasingly vulnerable to the choices software development organisations make.
Every day we hear of a new data leak. Confidential information is stolen and
sold to the highest bidder. Lives are meddled with and lives are ruined. User
facing software upon which people rely to make a livelihood breaks and those
users shoulder the burden of the lost time or inconvenience. In extreme cases
people die. Virtual bridges are collapsing around us and the best we can say as
a profession is that such collapses are a cost society is willing to accept in
the name of exponential progress.

Safety, is not our first priority.

What if safety were our first priority?

Engineering failures threatening the safety of others can be avoided so long as
we are willing, as a profession, to make certain trade-offs. If you ever wonder
why the pace of change for aviation, rail transportation and civil architecture
seems to be glacial in comparison, it is because safety is the first priority.

If we want to make safety the first priority, there are some uncomfortable
counter-realities that we may need to accept. It is plausible that in such an
alternate timeline, the smartphone may have never been brought to the mass
market, It is possible that social networking and the writeable web would have
been considered and then quietly abandoned, due to the inherent risk associated
with storing confidential information about millions of people in a central
repository. Your operating system and applications may be considerably more
stable, but the redundancy required to avoid takings risks would mean that its
performance and scope of functionality would be heavily reduced.

These costs, are, I suspect, too much for our cutting-edge society and cut-
throat business culture to bear. One thing I do believe though is that we can
still put the safety of users as our number one priority by focusing on the 20%
which will avoid 80% of the risk. We can, as a profession, still ask the hard
questions which might affect the bottom line. We can still ask whether a
particular feature, though innovative, might bring with it risks to the people
who depend on us that those people would find unacceptable, even if it meant
sacrificing the feature. We can still look at the systems we are designing and
ask "do I understand the impact upon a person if this fails?" We can still sit
down and make sure the math checks out before proceeding on a course of conduct
which may have irreversible consequences.

If we just sat down and accepted that certain trade-offs will have to be made,
we can be a profession that can be proud to say that safety is our first
priority.

Thanks to Wayne Spilsbury for providing feedback on and editing this essay

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s