The exceptionalism of software is unwarranted

If you are a trained engineer in an engineering discipline other than software engineering, entering the world of software exposes you to various “hot takes” by both programmers/coders who call themselves “software engineers” and actual software engineers with either experience or a formal education in software engineering.

In a forum thread regarding the excellent article “Why is this hard?” by James Edward Gray II I came across a few of those takes the indicate a misplaced sense of “software exceptionalism” that I’ve noticed again and again since 2015, when I entered the world of software engineering and development while at Hilti.

This is a summary of my own takes, informed by my mechanical engineering studies, my experience in the development and engineering of systems that are purely mechanical, multi-disciplinary (mechanical, electronics and software), and purely software, as well as service products and various business processes. It’s not my end-all-be-all view of things, but it’s where I stand after 24 years of experiencing and getting involved in different things.

How we develop a software system is different to developing anything else; but not that different

I worked for many years in companies developing mechanical and electronics products, and my take-away is that there is nothing special about software in that regard. The most important difference between software and other artifacts is its mutability and (relatively) much lower cost of engineering changes, especially in the early iterations, which carries over much better in the later iterations too!

Top-notch companies with people engaged in multi-disciplinary product development work also know by now to prototype early, and delay decisions until the gremlins have gotten worked out from the concept, the system architecture, and its design/implementation in hardware, whether that’s problems with turbine-blade resonances, casting process tolerances, RF interference, warping of parts during plastic injection molding, or durability of components due to vibrations. In those cases, once you “hit hardware”, even in prototype form, you are subject to supply-chain lead-times and loops that are far more expensive than the cost of the prototype’s production.

Having said that, there are certainly still many non-software businesses that develop in what many deride as “waterfall”–and even then, it’s practically never waterfall. It’s most likely Stage Gate / phase-gate, and even then, it’s untenable for such a business to launch a new product every 5-7 years as they once were used to. So, most non-software companies have already for years now adopted some ideas of agility (not mentioning capital-A Agility, that’s a whole different story, with rituals and certificates and other nonsense ), or introduced design processes and rules of thumb / internal guidelines for the design of key components.

Everything that has uncertainty has / must have a design process behind it, that reduces the uncertainty to an ALARP degree for the product to work as expected. From the “cartridge concept” of how some mechanical components (turbochargers, rotary valves are two I’m aware of) are designed to suddenly (after decades of design stagnation) prioritize repairability/maintainability, to figuring out the trade-offs behind the bounding box of the product, the size of the FPGA, the available space for the battery, etc., e.g. in this one that I was in charge of.

There is another take I’ve heard a few times going as far back as 2015: that developing software is fundamentally different to developing anything tangible, because it’s not “pure engineering”, and so it involves input and creativity at every step, instead of execution of tasks.

This take is absolutely, categorically wrong, and if you have indeed experienced engineering in this way, then you have my condolences.

The design process is part of the engineering process, and the engineering process lives and breathes exploration, creativity, and gathering input from all stakeholders of the system. Anything else, and you are not building a product, but a concept disconnected from implementability; i.e.: you are not engineering anything.

A design process decoupled from the engineering process produces mostly beautiful and mostly useless artifacts like the concept drawings that some industrial designers I’ve worked with come up with: beautiful forms, without any capability to implement some functions in the system, or severely compromising important requirements, or assuming specifications that are impossible to meet. Sure, Jim, you made it super duper pretty; oh the lines are so beautiful, the curves are so sexy! Too bad the battery that we can fit in the damn thing won’t last for even half an hour! But hey, the design is pretty and might win you one of the many awards handed out like bonbons (see also: Red Dot Design Award ) that you can put on your CV, I give you that!

The “software exceptionalism” regarding the process aspect is misplaced.

Estimation of software development work is different; but not that different

Estimation is similarly an issue in non-software businesses, too. Generally, pointy-haired bosses who don’t understand or don’t want to understand something that goes against what they’re being pressured to enforce, are what ruins things.

True story: I once was asked to develop the shaft size of new turbochargers. Eh, we have design rules for that, how hard can it be? Original estimate (by my decidedly non-PHB supervisor) was 3 weeks. Within these 3 weeks it became obvious that the design rules were useless, as they targeted shafts of single-stage turbocharging systems. My investigation for updating the design guideline with new rules for higher pressures and 2-stage turbocharging systems took 13 months. Good thing that it wasn’t on the critical path, and thanks to parametric 3D CAD we could delay decisions about the shaft’s interfaces and converge as late as possible.

Three months into the investigation the formulas for the first 2-stage system’s parameters were available (after millions of Monte-Carlo simulations), so no harm was done. But “3 weeks” vs. “3 months” is still a crazy difference. It’s what you don’t know that you don’t know that makes estimation tricky, and there is only one factor that makes this tricker with software (more on that below).

Making bold estimations without understanding what you’re really estimating is hare-brained, regardless of software or non-software work. And most people in managerial roles are far removed from the actual system, to know what they’re really asking for, or what they can expect, or what slack they should allow for.

The problem of estimation is not specific to the type of artifact being developed, but a “human problem”, regardless of discipline. It’s generally a good idea to “underpromise and underdeliver”, and to not commit blindly to targets before you know enough to judge them as realistic or unrealistic.

However, people often get carried away, especially the closer you get to getting an order, or being able to issue an invoice. The situation is function-specific, too. People in different business functions are clustered around different character archetypes. I could never be in direct sales and “knocking on doors” or “pounding pavement”, because I hate its stochastic nature, and the need for hyperbole that goes with it all too often… But others thrive on this as a source of excitement, and thus tend to over-overpromise, and then indirectly increase expectations on everyone else who’s tasked with the product, because management after all (generally) tends to take the idiotic “don’t bring me problems, bring me solutions” stance above the honest and inquisitive stance of “I don’t know, how can we figure it out?” that drives actual progress.

And that’s how the management of expectations also gets stratified politically throughout the organizational hierarchy… You can’t outplay those who rose through the ranks by being proficient at the game of managing expectations and promising upwards and dumping downwards. For an exploration of what happens if you try, see Lo and Behold, X! .

BTW, a very good book that more managers (and engineers!) should have read: “How to Measure Anything: Finding the Value of Intangibles in Business 3rd Edition” , by Doug Hubbard.

Programming/coding is not engineering

Some may claim that software engineering is different to engineering a non-software “thing”, because you are not engineering a physical product, but writing a set of instructions that tell the computer what to do. So, as someone aptly put it (a nice metaphor!), “we are closer to playwrights than engineers”.

This “playwright” lens is not describing an engineering process; it’s describing only a small part of it: coding (or programming). In non-software engineering, this is on a “level” of the V-model close to “implementation” (detailed design), similar to producing a CAD model, which are the instructions that tell a CNC machine or a machine operator (lathe, mill, sheet press, sheet-bending machine, etc.) what to do–i.e., the implementation.

And even then, the person in charge of producing the CAD model is (hopefully, but you’d be surprised) not a mindless, ignorant draftsman drawing lines in 2D or 3D without care, but someone who has enough knowledge of engineering to understand the trade-offs behind mechanical design decisions, such as stress concentration , engineering tolerances , or anything else that might impact how a line is drawn and how it connects to other lines.

The drawing is to the operator what the codebase is to the computer. That doesn’t mean that the process leading to the “implementation instructions” does not resemble engineering, regardless of the artifact being engineered.

I mean, of course you can go ahead and be a tinkerer, i.e. someone who goes to the junkyard, pulls things that seem they might fit together, and then kinda puts them together into something that might work or might not, depending on the system’s context and the properties of its components.

This is equivalent to copy-pasting code from Stack Overflow or from an LLM chat without actually understanding what it does, what it does not, what it does badly and when, and how it fails, and how its failure will impact the overall “system”. Vibe-coding and coding by copy-pasting is (usually, for most enthusiasts) the low-knowledge and low-understanding slapping-together-things-until-they-maybe-somehow-work-together equivalent of tinkering.

Engineering though requires understanding the system, its architecture, its components, how they connect, how they should work vs. how they work, why they work this way and not that way, under which conditions the components and subsystems/assemblies will work in isolation and together as you specified them, so that the system fulfills the requirements posed on it, when the components, subsystems/assemblies and the entire system is expected to fail and why (see FMEA). And the story goes all the way up both sides of the V, so that you understand what it is you’re building to begin with, and how you will know you’ve succeeded, and to what extent, and with which “gremlins in the system”. You need to make conscious trade-offs, minimize compromises, and avoid sacrifices , and you cannot do all that without engineering.

When you merely code, you might not be engineering the software, except if you’ve put forethought on the design that drives the code, the architecture that drives how everything “talks” to other things within the codebase, the requirements that impact the architecture you choose (from how you lay out your Elixir modules to whether you have microservices or a monolith or anything else), etc.

When you engineer the software, you have to build knowledge early enough, so that you can make better decisions. You don’t need to wait until the whole left branch of the V-model is “perfect”. You can experiment (e.g. in IEx) and build knowledge. But that’s no different to engineers 3D printing mock-ups, for example to test ergonomics, or running FEM analyses to figure out whether a requirement is realistic to begin with (“Can I make it as light as they think we should?”, “Can we bend that sheet how the industrial designer hoped we can without compromising drop-resistance?” and so on).

Every kind of engineering has its own modalities, such as the degree of mutability (and the cost of changes) of the artifact depending on how far along you are along the process from an idea to a prototype or a ready-to-launch product (or refinery, or defibrilator, or building, or anything else).

Software too; when you don’t just write a script but you engineer a software system (or a subsystem, like an Elixir module), you engineer it like any artifact beholden to any other engineering discipline(s).

Coding, i.e. “writing a set of instructions that tell the computer what do to”, is merely the implementation. “Coding” here literally means encoding a design into instructions. So, you cannot have instructions without this encoding, and therefore you can’t have implementation without some degree of design. And so, while coding you are in the engineering process already, but perhaps not all the way to the top of the left branch of the V in the V-model.

The program is not (always) the product; it’s a vehicle for achieving goals

The program (the code) is the design, or rather: the implementation of the design, though the relationship is bidirectional, as things usually are.

The program (the code) is the product only if you are in the business of selling the code itself; but not all software businesses are development-contract businesses. You can also sell pre-compiled binaries, and then you aren’t selling the design itself, but the equivalent of a widget (barring the ability to duplicate the widget with marginally zero cost). You could also be selling the UX and benefits that the design enables, e.g. if the product is Software-as-a-Service. But then, the program isn’t the only part of the design; how it’s been deployed and how it’s operated is also part of the product.

The code embedding knowledge is no different to designs embedding knowledge

Here’s another real-life story: a company I used to work for had launched a very successful tool more than 30 years prior. It established a new product category. I won’t reveal what it is to adhere to my NDA, even though it has long-since lapsed.

Over the decades, people had been promoted to other roles / other Business Units, product managers had long moved on, others across functions had left. The original set of people who had achieved a rich, nuanced understanding of all the decisions that had gone into going from the original ex-nihilo idea to a product that soon started getting copied by the competition, had been scattered to the four winds.

A friend was in charge of the product development project to develop a successor to that model, that had since undergone “facelifts” and minor upgrades, as the understanding of how this thing actually works had dwindled. She found out that the company had gotten rid of “old files in drawers”, and that included all the documentation around the project that defined the market opportunity, the project(s) that documented the development of the technologies and components the successful product utilized, the project that developed the product, etc. Drawings were available, obviously, as the widget kept coming off the production line.

But the knowledge had long been lost. They had to do a lot of reverse engineering and “theory-building” to get up to speed with the state of knowledge from decades prior. This is rework that could have been avoided if not for some “neat freaks” who didn’t understand the opportunity cost of throwing out old binders or wiping computer folders full of knowledge…

So, the “binary” (the product), as well as the drawings behind it (incl. specifications for parts production and instructions for assembly, maintenance and repair), were also “dead”, as they didn’t incorporate the “why” behind why this screws into that, why it should survive a drop from up to x centimeters height above ground, why something is like this instead of like that, etc.

How is it different to software? It isn’t.

In any case, for a business, product itself is always (should always be) a vehicle, even if it’s a very important vehicle. Of course, this will be different for different businesses; it all depends on how small or large the scope of the system you’re looking at is. For the subsystem of a business we call a team or department of developers, yeah, the instructions are the output, the end goal. For the larger subsystem called “product development” (department, team, process, however you organize things), the instructions are merely a vehicle to achieving some other end goal of that subsystem, that contributes towards end goals of the larger context etc. etc. which at some points ends with the “fruit” of the overarching system achieving something. That something might be “sustainable revenues”, “gaining market share”, “people find my Hex.pm packages useful”, etc.

The crux of the matter is that the world of software often mixes two things, by virtue (or vice) of its workforce not having formal or acquired experience in systems engineering: product development vs. coding. These two things are not the same! The former encapsulates the latter as an activity.

The code is the fruit of the design, and obviously incorporates design decisions along the way. Therefore, the code is the implementation of the design, and the computer is the “machine operator” / CNC that turns the instructions into the widget (an executable binary).

Development is theory-building, regardless of what you develop

Figuring out things as you go, iterating on the design of your modules, improving the system architecture and how its modules and components talk to each other, etc., is not special when it comes to software. When you engineer any system, you have to figure out and understand its trade-offs, so that you can make better, informed decisions that bring the product (and the project that delivers the product) closer to the goals (cost vs. budget, time to launch, revenue potential, distribution, post-launch needs, etc.), and minimize compromises and avoid sacrifices that would unnecessarily cause diversions from the goal(s).

This is also theory-building, regardless of the nature of the artifact being used as a vehicle for gaining customers, achieving profitable growth, etc. “Theory-building” is a fancy way of saying “closing the knowledge gaps”, which itself is a Lean Product Development consultant’s/expert’s fancy way of saying the much more pedestrian and obvious “don’t make uninformed decisions”, because uninformed decisions increase the risk of the endeavor.

Beyond mutability, non-adherence to unbreakable laws is what makes software different

Compared to software systems, non-software systems are subject to the unbreakable laws of physics. Yes, ultimately, the code you write ultimately compiles down to code that transistors work on, which are also subject to physics. But the number of layered abstractions stacked on top of each other from going to semiconductor physics to how you make a button react to clicks is so high, that by the time you are at the highest level of abstraction, any physics constraints are long since ignorable.

This is a big difference between software and non-software systems. In mechanical engineering systems, for example: sure, we use simpler models and even CFD/FEA to build abstractions with predictive power by modeling laws of thermodynamics, etc. However, a key difference to software is that we are closer to “bare metal”. We have far less choice of the system architecture, because our abstractions are lower-level, and the interfaces are tangible, not conceptual.

This difference, combined with the insane degree of mutability of the design of software compared to the design of other types of artifacts, is what makes estimation more tricky in the world of software. Between requirements and design, let alone implementation, in software there is a much bigger “cloud of uncertainty” compared to other, non-software artifacts, exactly because of the abstraction level of the systems being developed. You can achieve the same results in so many different ways (not all of them equally good) that you can come up with estimates of anything from 1 hour to 1 month for a new task/feature/bugfix, depending on how many abstraction levels and system interfaces you choose to smash on the path to a solution.

Software-venture managers who don’t understand this make things even more difficult than they need to be, as they often don’t ask for the trade-off between estimated effort and extent of mutations along the way; instead, they employ “Agile shamans” who ask for some silly thing like “T-shirt sizes” or “story points” that are then fed into some Jira or other software, in the process also reducing the number of options doing to a single one, pretending that there is one way to do it, that the estimate is not an estimate but some kind of promise or contract. The Pikachu face is guaranteed, especially when you see the joint probability of all estimates being accurate.

So yes, it’s more tricky to estimate effort and lead times in software. But mostly because people don’t open up the funnel of options, don’t evaluate different options with regards to different metrics (number of interfaces to be modified depending on the solution option, for one), and then narrow things down to single-point estimates, leading to uninformed decisions that–of course–make risk explode. See also: “What the coastline paradox reveals about infinite software specialization (in martech and other fields)” .

Bad requirements? Bad implementation.

Anyone who has worked on tangible products from widgets to houses can attest to this: if the requirements are unclear, then the specifications are likely to include errors, and no matter how pretty the design, if the actual ground-truth requirements are not being met, the value proposition of the product cannot be fulfilled.

This is no different in the world of software. This is about defining goals, eliciting functional and non-functional requirements (including company-internal and legal ones), aligning them against the project plan, its viability/ROI and the related business plan, etc.–not very different to developing anything, software or not.

Anything that involves humans involves subjective opinions, and the more subjective the subject-matter, the more you have to navigate the process politically.

The upside of working on tangible products is that senior stakeholders can beat their chest all they want, but the laws of physics don’t care. Whereas, in software some PHB will come up with “requirements” stemming from chasing fads like “can we use GenAI for that?”, “of course we’ll deploy on Azure”, “of course we’ll use microservices, I read about this in the McKinsey Quarterly!”, “the project must do [sic] Agile” etc., and this throws a wrench in things…

Blessed are those who don’t have to work under the whims of idiots chasing the shiny new thing!

Engineering is inevitable

Developing software is certainly different to developing anything else, as developing anything is different to developing anything else, even within any specific engineering discipline; developing a bridge is different to developing a house, after all. Same goes for developing a turbine compared to developing a Ground Penetrating Radar. However, the differences are overstated, and this makes software not that exceptional. The differences can be smoothed over by applying an engineering approach on software too.

In my opinion, the world of software would benefit from more engineering-minded approaches. We would have better products and user experiences, if we were to reduce our reliance on wishful-thinking that we can develop robust systems without having to deal with understanding their trade-offs, such as the growing trend of slapping things together with minimal understanding that is so en vogue due to the rise of vibe coding.

How to achieve this without regressing back to the world of “requirements monkeys”, bureaucratically-driven “Requirements Engineering” processes with Rational DOORS and hundred-page Big-Design-Up-Front documents, and all that ancient cruft is a different problem, but not an intractable one .