As our dependence on technology increases, the design of software matters more than ever before. Why then is so much software flawed? Why hasn鈥檛 there been a systematic and scalable way to create software that is easy to use, robust, and secure?
Examining these issues in depth, The Essence of Software introduces a theory of software design that gives new answers to old questions. Daniel Jackson explains that a software system should be viewed as a collection of interacting concepts, breaking the functionality into manageable parts and providing a new framework for thinking about design. Through this radical and original perspective, Jackson lays out a practical and coherent path, accessible to anyone鈥攆rom strategist and marketer to UX designer, architect, or programmer鈥攆or making software that is empowering, dependable, and a delight to use.
What inspired you to write this book?
DJ: As a user of software, I鈥檇 always found it fascinating that some apps were so good,and others so bad. Even within one company鈥檚 products, you come across one app that just seems so right in every respect, and another that鈥檚 falling apart at the seams. I鈥檇 been a computer scientist for 25 years, and I decided that the time had come to apply everything I鈥檇 learned to figuring out what was going on, in the hope that I could find a way to get things right.
How long did that take, and what did you find?
DJ: It鈥檚 been almost 9 years to the day since I submitted a proposal to the National Science Foundation for a project to explore the idea that concepts were the essence of software design, claiming that if we could figure out what they were, we would be able to improve our software greatly. The reviewers said that 鈥渢he proposal does not provide a sufficiently detailed plan for the panel to assess the possibility for success鈥� and duly rejected it. It wasn鈥檛 an unreasonable criticism. I really had no idea where this project would go.
So I spent the next nine years taking careful notes every time an app did something confusing or wrong, and slowly but surely building a theory that could explain all these flaws. My book presents a sample of the best of these flaws鈥攖he most illuminating and occasionally amusing鈥攁nd I was amazed to find (counting the entries in the index of apps recently) that 100 apps are mentioned.
What is this theory?
DJ: In short, it鈥檚 a way to think about any software product, from the smallest app to the largest system, as a combination of 鈥渃oncepts.鈥� People in computing have talked about concepts seemingly forever. Fred Brooks, in Mythical Man Month, says that conceptual integrity is the key to success in software development. And the whole field of usability rests on the idea that the user鈥檚 conceptual model should match the designer鈥檚 model. But nobody had ever tried to figure out what a concept is (in the context of software apps, at least), or what it means to have conceptual integrity.
So what is a concept?
DJ: First, let me say what it isn鈥檛. There are lots of contenders for concepts in computer science, such as abstract data types, classes, objects, concept lattices, entities, and so on. But I couldn鈥檛 make any of these work. In short, I discovered that a concept is really a little behavioral protocol. Think of the real-world concept of a restaurant reservation. If you had to explain that to a Martian (who we鈥檒l suppose already knows what a restaurant is), you wouldn鈥檛 talk about structural relationships or classifications. You鈥檇 say something like: it鈥檚 to make sure that when you turn up at the restaurant, they have a table for you, and the way it works is that you call in advance and agree on a time and then later turn up at that time. This explanation has two key parts: a purpose (explaining what a reservation is for), and an operational principle (explaining the basic behavioral pattern). In my book, I show how this framework can be used to identify and define as concepts all the kinds of functionality that arise in software.
Is this philosophical or practical?
DJ: My goal was primarily to develop an approach that can be readily applied, so it鈥檚 very practical. Every idea in the book is introduced with a practical example, and there are practical lessons highlighted throughout. It鈥檚 also a bit philosophical, because if we want to make radically better software, we need to think about it differently too. Don鈥檛 worry though; I鈥檝e relegated the more cerebral discussions to the endnotes, along with some anecdotes that were a bit too long to include in the body, and a bunch of other fun things, including the recipe for my favorite pasta sauce.
How does this help you design better software?
DJ: First, it gives a new framing, so you can organize your design and think about it with greater clarity. Second, the theory includes a collection of design principles that you can apply to help you discover problems early on鈥攑roblems that you might not have found in user testing but which could turn out to be very serious. And third, concepts help you realize when you鈥檙e reinventing the wheel, by capturing established design ideas in reusable form.
Are these reusable concepts like software design patterns?
DJ: They鈥檙e more like Christopher Alexander鈥檚 design patterns in architecture, because they apply at the level of what the user experiences, and help you organize the design at the highest level, whereas software design patterns focus on structuring object-oriented code. But the theory of concepts is influenced by software design patterns, and incorporates many other great ideas from software research.
How is concept-design related to design thinking?
DJ: Design thinking is a collection of design practices, with a focus on the process鈥攚hat steps you take, and in which order. It emphasizes early prototyping, and moving back and forth between 鈥渄ivergent鈥� phases, in which you brainstorm and expand the list of ideas, and 鈥渃onvergent鈥� phases, in which you narrow your design down.
None of this is software specific, and it鈥檚 not clear how to apply it to a big software problem. You can鈥檛 ideate effectively about an electronic health record system or even a moderately complex app. To apply design thinking to software, you first need some way to break the problem into pieces that can you can think about independently. That鈥檚 one respect in which concept design fits in.
Another is that design thinking doesn鈥檛 have much to say about reusing design ideas, even though (of course) the best designers do it all the time. Concept design gives you a framework for encapsulating existing design ideas and finding opportunities to reuse them. Imagine, for example, designing a new social media app. You might not want to reinvent standard concepts like 鈥減ost,鈥� 鈥渃omment,鈥� 鈥渓ike,鈥� 鈥渒arma,鈥� and so on. With concepts, you can include those in your design as well-understood components, freeing your creativity up to invent new concepts (and find novel ways to assemble old concepts together).
Have you read Don Norman鈥檚 鈥淒esign of Everyday Things鈥�?
DJ: It鈥檚 a wonderful book and was a major inspiration. I modeled my book on it in several respects, especially in the way I use examples to drive the discussion, and in casting the contribution as a simple theory. In some ways, my book is an attempt to do for software what Don Norman did for physical things (especially doors!). In his design theory, there is a key assumption that the actions that users perform have immediate effects: you pull the door and it opens; you flick the switch and light goes on; etc. His central idea of affordance builds on this assumption: namely that each control should signal the effect it has to the user. As software has become more complex, however, most of the effects of user actions are only felt much later. For this reason, I believed that a different kind of theory was needed that could explain the chain of events connecting the user鈥檚 action (tagging a photo in Facebook, for example) to an outcome that the user might not see immediately (making the photo visible to a much enlarged group of users).
Does concept design impact how code is written?
DJ: Yes, because it offers a much more refined kind of modularity. The implementation of a concept can be thought as what programmers call a 鈥渟ervice,鈥� but concepts are smaller than traditional services, even than microservices, so I like to call them 鈥渘anoservices.鈥�
Isn鈥檛 software today pretty good already?
DJ: Yes, a lot of it is really great. But there are still flaws, and as we spend more and more of our time using software, the flaws become more annoying. And as we rely more and more on software, the economic and social costs of software flaws become more apparent.
In my book, almost all the examples are drawn from apps built by the companies that produce the best software, (Apple, Dropbox, Facebook, Google, Microsoft, etc), because I didn鈥檛 want people to say I was picking soft targets. Focusing on these apps also allowed me to check that the flaws I identified really mattered. When I found a flaw in my investigations, I would scour the web to find reports from users, and inevitably came across sad stories about how the flaw had led to some irreparable harm. In one of the examples I discuss in the book, a user was so upset that he registered a domain name protesting the app鈥檚 failings.
But the lessons of the book are even more applicable to the software produced by companies that have less experience and fewer resources. This kind of software has fewer users than apps like Gmail, but sometimes it matters even more. Most companies rely on enterprise software, for example, that is of much lower quality, and medical devices (whose failures causes tens of thousands of deaths every year) are often rife with design flaws. In general, the software that runs in consumer appliances (cars, cameras, ovens, printers, etc) is not yet built to the standard of the best commercial software, and I believe could benefit greatly from the ideas in this book.
What鈥檚 your favorite example of a concept?
DJ: I think it would have to be the trash concept. I feel a bit nostalgic about it, because it was the first concept I identified and described carefully. It has a long and noble history, being the very emblem of the Apple Macintosh and its usability from its earliest days, and it was central to a major court case between Apple and Microsoft. The idea of the trash concept was so good that it鈥檚 now become ubiquitous, and appears in many applications and desktop file managers.
The way in which the trash is treated as a folder in the Macintosh is an example of what I call 鈥渟ynergy,鈥� which I explain in the book harder to achieve than it seems, and still shows some rough edges. In fact, the trash is a great example of a concept that has evolved over a long period of time. Even in the last couple of updates of macOS, there have been changes that have resolved some longstanding issues. And a few still remain: it is possible, for example, to empty the trash and find that no storage space is reclaimed at all, even if the trash previously contained some huge files (see note 73!). In summary, the trash exemplifies for me how far we鈥檝e come and how far we still have to go.
Daniel Jackson is professor of computer science at the Massachusetts Institute of Technology, and an associate director of the MIT Computer Science and Artificial Intelligence Laboratory. He is the author of Software Abstractions and Portraits of Resilience.