The Hidden Saboteurs of Software Engineering, Part One: Documentation Practices
How We Unwittingly Sabotage Our Best Efforts
Introducing the Field Manual
In 1944, the United States Office of Strategic Services (OSS) published the Simple Sabotage Field Manual, teaching subtle behaviors to disrupt enemy organizations. Rather than dramatic acts, the manual advocated for seemingly innocent activities that would disrupt an organization by overwhelming the system's ability to function effectively.
Today's software development practices often unwittingly implement these patterns of disruption. Many of these practices can be considered self-sabotage. That is, we are not enduring sabotage by external forces, such as competitors or regulations, but rather by ourselves, making us our own worst enemy. Walt Kelly, in his famous comic strip Pogo, aptly captured this phenomenon in a telling quote.
“We have met the enemy and he is us.”
This article is one of a series and covers a pervasive form of simple (self) sabotage: documentation practices, described by the field manual as “Haggle over precise wordings of communications, minutes, resolutions.”
While reading this paper, it may be helpful to consider the following questions.
Overarching questions to consider:
What are we doing that’s self/simple sabotage?
What alternatives might address the dynamic nature of software development more effectively?
Questions to consider in regards to documentation:
Are the models we have about the value of documentation based on fundamentally flawed premises?
If we know that documentation is inherently flawed, can we better manage knowledge in software development?
How might we shift to recognize these limitations while still benefiting from positive aspects of documentation?
Sabotage Tactic: “Haggle over precise wordings of communications, minutes, resolutions.”
What We Think We’re Optimizing For
Documentation is often perceived as a means to optimize communication across teams. It’s believed to capture diverse opinions, improving decision-making ability by leveraging collective knowledge. It’s typically seen as a way to build consensus, leading to timely, informed decisions. The underlying assumption is that paying meticulous attention to documentation delineates individual responsibilities, creating an environment where work can proceed efficiently.
It's often believed that well-crafted documentation prevents ambiguities, reducing the errors and misunderstandings that lead people to do the wrong thing. This belief assumes that thorough documentation optimizes for perfect understanding and seamless collaboration.
What This Actually Leads To
Haggling over document details results in a number of problems. Each is discussed in the following sections.
Decision Queueing and Inventory
Haggling over document details inevitably delays decision-making. In lean philosophy, delayed decisions can be considered inventory, which is viewed as waste.
The problem compounds when this decision inventory goes into queues. As with inventory, in lean philosophy, queues are waste, creating multiple cascading layers of waste. Essentially, decisions are put on hold while debating document specifics.
Effective systems generally exhibit a bias to action over perpetual discussion, which assists decision-making. While delaying decisions may help gather additional information, excessive delays hinder progress. With delayed decisions, we inadvertently mirror the field manual's strategy of creating bottlenecks.
Decision Fatigue/Hierarchy
Decision fatigue occurs when facing an overwhelming number of choices. These choices lead to diminished decision quality over time, the accumulation of unresolved issues (decision inventory), and increasing decision dependency. As multiple decisions are debated, mental ability deteriorates, mirroring the manual's strategy of reducing system effectiveness through overload.
Decisions in software development often form a recursive or hierarchical structure. Delays in one decision cascade outward, impacting dependent decisions and causing a chain of delays throughout the hierarchy. These delays degrade decision quality, accrue decision inventory, and deepen dependency chains. Consequently, related decisions are paralyzed, slowing progress, and acting as another silent saboteur in our operations, supporting the field manual's tactics of overwhelming systems in an attempt to reduce their effectiveness.
Pitfalls of Using Documentation as a Proxy for Collaboration
When teams substitute documentation for direct collaboration, effort shifts from genuine alignment via teamwork to proxy communication via documentation. By separating the people who should be working together, we introduce problems that would not otherwise exist.
Documentation models the current state of knowledge at a given point in time. However, understanding of the work to be done continuously evolves. This fundamental mismatch creates an ongoing maintenance burden. Frequent meetings become necessary to synchronize documentation with evolving understanding because documentation accuracy degrades with time. When left unaddressed, outdated documentation can become actively harmful. Additionally, even after meetings, documentation often fails to capture the full context and depth of knowledge of what must be done.
Another pitfall is a misplaced focus on documentation, resulting in prolonged debates over format and content, diverting attention from customer-centric work. This preoccupation with form over substance leads to visually appealing documents that fail to convey the customer's voice.
The cycle of frequently halting work to update documentation disrupts the flow of valuable software, adding inefficiency and waste that compounds over time, eventually making us ineffective.
Documentation completion doesn't ensure its accuracy. Validating documented decisions remains hidden in an invisible queue until the actual work reveals knowledge gaps and miscommunication. Feedback delays are an inherent trait of documentation-based communication, preventing rapid testing of ideas in the real world. [1]
Errors compound with time when we delay feedback on our ideas. The harm caused by this typically isn't discovered until work is completed, necessitating costly and time-consuming rework.
The practice of investing time in inherently flawed documentation while delaying real-world validation exemplifies the field manual's approach to system disruption.
The Documentation as Scapegoat
Documentation errors create pathways to blaming and scapegoating. When work is undertaken and later discovered to be wrong, it's easy to point fingers with statements like "Don't blame me. It's not my fault. I did what the document said."
In punitive environments, where self-preservation takes precedence over customer value, blaming and deflection become normalized. People are incentivized to protect themselves first and delight the customer second. In these environments, documentation takes on outsized importance, at times becoming more important than the software itself.
Software failures stem from complex issues that include miscommunication, inadequate testing, and flawed design. However, documentation's simple, tangible nature makes it an appealing target, leading to a simplification of what are often complex failures. This simplification leads to the mistaken belief that we can avoid failure with sufficiently accurate documentation. We enter a feedback loop of perpetually failing software whose "remedy" is better documentation.
In effect, we believe documentation is the panacea that will cure our software ills, failing to see that no amount of effort can make it infallible. By nature, documentation is flawed and insufficiently representative of the work we must do.
Ultimately, scapegoating and blaming don't produce work products that delight the customer. Instead, they divert us from the work we must do.
Fear of Working With Others
Emphasizing documentation as our communication mechanism provides a refuge for those who fear direct collaboration, acting as a shield against vulnerability. This fear sometimes arises from being "discovered" as imperfect and fallible.
Misused this way, documentation helps us avoid the burden of "proving our worth" when collaborating closely with our colleagues. It's a common belief that when working alongside others, we must have the answers and must be correct, thereby proving our value.
We often believe that if we perfect our documentation, we can work independently, falsely believing that independent work is the most efficient method. [2]
"Perfect" documentation allows us to hide behind the illusory certainty of written words rather than engage in the messy, imperfect nature of collaborative work.
Summary
Using documentation to communicate seems to offer improved alignment, decision-making, knowledge sharing, and efficient work management. These assumed benefits only partially materialize. Instead, Haggling Over Documentation creates systemic inefficiencies that mirror the field manual's disruption tactics.
The following section discusses ideas of what we might consider instead.
What To Consider Instead
It's important to state at the outset that context matters. Each of us must find our own way forward, unique to our company and its circumstances. With that in mind, the following are several principles that merit consideration.
Rethinking Documentation’s Role
The misguided goal of keeping documentation up to date inhibits us from exploring better ways to obtain and preserve knowledge. What if documentation could serve as a guide for principles and high-level concepts rather than an authoritative source of detailed specifics? We might then accept its limitations, allowing for more dynamic knowledge diffusion and preservation methods. Instead of perpetually updating documentation, the focus could shift to accruing knowledge in a constantly evolving environment. Further, we might then rethink the belief that documents are error-free and static once "finalized," understanding that they can't capture the full context of our knowledge.
Prioritizing Collaboration Over Documentation
Software development teams often face a choice between emphasizing documentation and focusing on direct collaboration and delivery. Software Teaming (Mob Programming), where teams work together on the same task simultaneously, exemplifies a collaborative approach and keeps knowledge current through active engagement rather than documentation. [3]
Unlike static documentation, whose accuracy decays between meetings, real-time collaboration keeps pace with evolving knowledge. By contrast, a document-centric approach imposes a heavy administrative burden via frequent meetings to keep documents aligned with the current state of knowledge.
Emphasizing Flow Over Utilization
Most organizations focus on maximizing worker utilization, viewing idle workers as the primary source of waste. However, as Donald Reinertsen observes in The Principles of Product Development Flow: "In product development, our greatest waste is not unproductive engineers, but work products sitting idle in queues." [4] This insight challenges traditional efficiency metrics that prioritize keeping everyone busy and make it unnecessarily hard to get work done. Understandably, it may be difficult to entertain the idea of workers not always being 100% utilized, but considering the cost of queues may make it easier to accept.
While intended to coordinate work and maintain alignment, documentation-heavy processes create queues for review cycles, handoffs, and approval gates, wherein each document becomes another work product waiting in line, adding to the growing pool of idle inventory.
Consider focusing on keeping the work flowing toward the customer, where value lies, instead of the maximum utilization of everyone. [2]. Put simply, organize for delivery rather than "productivity" and "efficiency."
Direct collaboration may lower individual utilization rates, making it more difficult for people managers, but it improves effectiveness, which is ultimately a more helpful metric. When focusing on effectiveness, we maintain a continuous flow of value to customers, the true measure of system efficiency.
In summary, documentation-heavy approaches attempt to capture collective knowledge, team alignment, and a roadmap for work but often fall short. By contrast, a collaborative approach focuses on keeping the work flowing, minimizing the need to "haggle over precise wordings of communications, minutes, resolutions."
References
[1] Meadows, J., 2023, “The Siren Song of Software Planning”
https://medium.com/@jmlascala71/the-siren-song-of-software-planning-237f1feab133
[2] Meadows, J., 2023, “Utilization Considered Harmful” https://medium.com/@jmlascala71/utilization-considered-harmful-f992776e5e3e
[3] Zuill, W. and Meadows, K., 2022, “Software Teaming, A Mob Programming, Whole-Team Approach. Second Edition”
[4] Reinertsen, Donald G., “The Principles of Product Development Flow: Second Generation” Lean Product Development. Celeritas Publishing, 2009.
 | A guest post by
|