“Leaky” Abstractions and Conway’s Law: The Reality of Development

How “leaks”, redundancy, and team structure shape software.

Executive Summary

Abstractions simplify complex software systems but are inherently imperfect, as per Joel Spolsky’s Law of Leaky Abstractions, which states that all non-trivial abstractions leak, causing unexpected issues like performance or security problems. Developers must understand underlying details to address these leaks effectively. Conway’s Law further complicates development, as system designs mirror organizational communication structures, often leading to suboptimal architectures due to team dynamics.

Key Points:

• Leaky Abstractions: Abstractions hide complexity but can fail, requiring developers to grasp implementation details to fix issues, optimize performance, or ensure security.
• Risks of Excessive Abstraction: Over-layering increases complexity, reduces performance, and complicates debugging, leading to technical debt and higher maintenance costs.
• Conway’s Law: Organizational structures shape system designs, with poor team communication resulting in rigid, inefficient interfaces and costly refactoring.
• Professional Skills Needed: Developers need mental flexibility, informed decision-making, and awareness of trade-offs to balance simplicity and performance while navigating organizational constraints.
• Recommendations: Use abstractions strategically, align architecture with team structures, prioritize simplicity, and invest in deep technical knowledge to make informed trade-offs for performance, scalability, and maintainability.

Mastering software development requires navigating leaky abstractions and organizational influences with flexibility and strategic decision-making. By balancing simplicity with control and aligning designs with team dynamics, developers can build robust, efficient systems, minimizing technical debt and ensuring long-term success.

Introduction: A World Built on Abstractions

We live and work in a world of incredible complexity. From the microchips in our smartphones to global logistics networks and intricate software systems — understanding all the operational details of even a single element of modern life is practically impossible. This is where one of the most powerful tools of human thought and engineering practice comes to our aid — abstraction.

At its core, abstraction is a process of simplification where we hide currently unnecessary implementation details, focusing only on the essential characteristics of an object or system. It’s a way to highlight the main points, to create a model that describes what something does, not how it does it. We ignore the internal workings to be able to interact with something at a higher, conceptual level.

Imagine a car: to drive it, you don’t need to know the thermodynamics of the internal combustion engine or the algorithms of the anti-lock braking system. You interact with abstractions — the steering wheel, pedals, gearshift — which hide colossal mechanical complexity beneath them. The same goes for a smartphone: we easily launch apps by touching icons, take photos with a single tap, or scroll through news feeds, completely oblivious to the workings of the operating system, processor architecture, data transfer protocols, or the physics of the touchscreen. All these highly complex processes are hidden behind a convenient and understandable interface — a set of abstractions. Abstraction turns an elephant, impossible to eat whole, into a set of convenient “black boxes” with understandable interfaces.

Abstractions are not just a convenience; they are a fundamental necessity for the functioning of the modern world. Without them, we would be paralyzed by complexity. Imagine if, to send an email, you had to delve deep into understanding how data packets are encoded, how routing protocols work, and the physical principles behind signal transmission over fiber optics every single time. Or if buying coffee required understanding the entire coffee bean supply chain, including agricultural subtleties, details of international trade agreements, and financial transaction mechanisms. We simply couldn’t cope with such an information flow and the constant need to grasp the implementation of every process.

Abstractions allow us to:

• Manage complexity: By breaking down huge systems into components that can be interacted with without holding their entire internal structure in mind.
• Specialize: Experts can deeply understand their narrow field (e.g., developing a graphics card driver), providing others with a simple abstraction (an API for rendering graphics), freeing them from needing to understand all the hardware nuances.
• Reuse solutions: We can use ready-made “black boxes” (code libraries, standard components, services), trusting their interfaces and not spending mental effort studying their implementation each time.
• Accelerate progress: We build new, more complex systems on top of existing abstractions, not requiring every developer or user to fully understand all underlying layers. From operating systems and programming languages to economic models and legal concepts (like a “legal entity”) — these are all levels of abstraction that structure our reality and make it manageable, freeing our minds from the need to constantly delve into implementation details.

Essentially, all of modern civilization, technology, and even our daily interactions are built on countless layers of abstractions that allow us to act effectively without drowning in the endless details of how everything works. They make the world not only possible but also convenient to use, allowing us to focus on our goals rather than the mechanisms for achieving them.

Abstraction is the mental process of identifying the essential properties of an object or phenomenon and disregarding the non-essential ones. It is a fundamental mechanism of our thinking, allowing us to form concepts, classify objects, and build generalized models of the world. We see a specific tree, but we form the abstract concept of “tree,” discarding the details of the specific instance (height, trunk thickness, number of leaves).

Moreover, we constantly operate with levels of abstraction. We can talk about a “living being” (very high level), a “mammal,” a “dog,” a “Labrador,” or “my Labrador, Rex” (very low, specific level). Each subsequent level adds detail and specificity, while each preceding level generalizes and simplifies. This mechanism allows us to think effectively about complex hierarchical systems, switching between “close-up” and “overview.”

In technology and engineering, we use the same principle. We create layers of abstraction, where each upper layer uses the functionality of the lower layer as a given, without delving into its internal workings.

So, for the average user, such multi-level abstraction makes the complex world simpler and more accessible. It presents an understandable interface, a “black box” that allows achieving a goal while hiding all the complexity under the hood. We press a button — we get a result, and we don’t need to know what happens inside.

However, for the professional — the engineer, the developer, the systems architect — the picture is much more complex and less rosy. Ideal, fully isolated layers of abstraction exist mostly in theory. In practice, professionals constantly encounter the fact that:

• Abstractions “leak”: Details that the abstraction was supposed to hide often have unexpected effects on the system’s operation — its performance, reliability, or behavior in non-standard situations. Relying entirely on the perfection of the “black box” turns out to be risky.
• Excess abstractions hinder: Attempting to hide complexity by adding more and more layers of abstraction can have the opposite effect: the system becomes convoluted, inefficient, and finding the causes of errors turns into navigating a labyrinth of layers. Simplification at each individual level breeds complexity overall.
• Communication structure dictates design: Conway’s Law is relentless — how teams are organized and communicate directly shapes the architecture of the systems they create, often leading to technically suboptimal solutions.

This story is dedicated precisely to this more complex reality. We will examine why abstractions are not always as ideal as they seem, how their “leaks”, redundancy, and organizational factors affect development, and why it is critically important for a professional to be able not only to use abstractions but also to “see through them,” understand their limitations, and manage the inevitable complexity they are designed to hide.

The Central Problem: The Law of Leaky Abstractions

The idea that abstractions can be imperfect crystallized into the concept known as The Law of Leaky Abstractions, popularized by Joel Spolsky. The law states:

All non-trivial abstractions, to some degree, are leaky.

This means that no abstraction created to simplify something complex (be it a software interface, a network protocol, a hardware component, or even an organizational structure) can completely hide all the details of the underlying implementation. Sooner or later, these hidden details “leak” out and become important for the correct use or understanding of the abstraction itself. And the more complex the system we are trying to abstract, the greater the likelihood and significance of these “leaks”.

Simply put, the ideal “black box” that can be used without any thought about its internal workings is more the exception than the rule, especially in complex systems like software.

When we say an abstraction “leaks”, it means that hidden implementation details start affecting how we must use that abstraction or how it behaves in real-world conditions. In other words, what was supposed to be invisible and unimportant to the user of the abstraction suddenly becomes visible and important.

This manifests in various ways:

• Performance impact: Many programming languages, including Python, have a simple and intuitive operation for concatenating strings — the + operator. Writing result = string1 + string2 + string3 seems natural. This operation is an abstraction over joining sequences of characters. However, strings are often implemented as immutable objects: once created, a string cannot be modified. Because of this, when you use the + operator to gradually build up a string, especially inside loops (final_string = final_string + next_part), the system is forced to create a new string object in memory each time and copy the data from the old string and the added part into it. If you concatenate hundreds or thousands of parts this way, it leads to a huge number of temporary objects and multiple data copies, making the code unexpectedly slow. To write efficient code, the developer needs to know about this “leak” in the + abstraction and use other methods specifically designed for this purpose (e.g., the join() method in Python for joining a list of strings), which work much more efficiently under the hood, avoiding unnecessary copying during bulk concatenation. The simple and convenient string addition abstraction has “leaked” and revealed its inefficiency when used in a specific, albeit common, scenario.
• Unexpected resource consumption: An abstraction might hide details of memory management, network connections, or other resources. Ease of use can mask the fact that numerous connections are being opened and not closed under the hood, or that vast amounts of data are being loaded into memory.
• Specific errors and failures: An abstraction might try to hide lower-level errors (like network failures) but does so imperfectly. As a result, the user of the abstraction encounters strange hangs, timeouts, or receives error messages that make no sense at their level of abstraction but are explainable if one knows the internal workings.
• Need to understand internal logic for correct usage: Sometimes, to use an abstraction effectively or correctly, one must understand certain aspects of its implementation. For example, to write fast code using a particular library, you need to know which of its functions make expensive system calls and which operate only in memory.
• Edge cases and limitations: An abstraction might work perfectly in 99% of cases, but in rare, edge situations (e.g., working with very large files, during peak loads, with specific input data), its behavior changes dramatically due to limitations of the underlying implementation.

Essentially, a “leak” means that the boundary between the abstraction and reality is not hermetic. Reality knocks at the door, and it cannot be ignored forever, even if the abstraction promises otherwise.

Why are these “leaks” inevitable? Why can’t abstractions be perfectly sealed containers?

Firstly, the reason lies in the fundamental complexity of the real world. The systems we try to abstract — be they physical processes, networks, operating systems, databases — are incredibly complex in themselves. They have numerous states, interactions, and nuances. Attempting to create a simple model (and the goal of abstraction is precisely simplification) for such a complex system inevitably means that some aspects will be omitted or represented inaccurately. Sooner or later, the unaccounted aspects of reality will make themselves known. Recall the TCP/IP abstraction, which promises reliable data delivery. It cannot completely hide the reality of unreliable networks — packets can be lost, delayed, or duplicated. The protocol tries to compensate for this, but the very necessity of these mechanisms and their limitations “leak” upwards in the form of delays and potential failures.

Secondly, leaks arise due to the inevitability of edge cases. Abstractions often work well under “normal” conditions for which they were primarily designed. But the real world is full of edge, rare, unexpected situations: null values, empty lists, huge amounts of data, peak loads, hardware failures, errors in input data, concurrent access to resources. Creating an abstraction that elegantly and correctly handles all possible edge cases is extremely difficult, and sometimes impossible without significantly complicating the abstraction itself. Often, it is these rare cases that cause the abstraction to “break” or behave unexpectedly, exposing implementation details.

Finally, abstractions “leak” due to design compromises. Creating any abstraction is always a balancing act. Developers are forced to make trade-offs between ease of use, functional completeness, performance, flexibility, and development cost. To make an interface simple, one often has to sacrifice the ability for fine-tuning or handling rare cases — these unaccounted possibilities then “leak” in the form of limitations. As we saw with the string concatenation example, a simple interface can hide very inefficient operations underneath for the sake of user convenience. To make it fast in all scenarios, the interface itself would have to be complicated, or the user would need more knowledge about its internal workings. Sometimes an abstraction tries to hide something that is fundamentally unhideable — like the finite speed of light in network protocols, which inevitably leads to delays, or the need for synchronization mechanisms during concurrent data access.

Essentially, “leakiness” is not so much a design flaw (although poor design can exacerbate the problem) as an inherent property of trying to impose a simple model onto a complex reality full of surprises and requiring constant compromises. A perfectly hermetic abstraction would require either infinite complexity, indistinguishable from reality itself, or operation in a completely predictable and limited environment, which is rarely encountered in practice.

Theoretically, a perfectly hermetic abstraction could exist, but only in a completely predictable and extremely limited environment. Imagine a system with no failures, no performance variability, no unexpected input data or complex interactions, running on perfect hardware. In such an “ideal world,” an abstraction could accurately reflect all the behavior of the underlying system, needing no compromises and having no hidden details capable of “leaking.”

But the reality of software development is infinitely far from this ideal. We work with hardware that can fail and have variable performance. We depend on operating systems with their background processes and limitations. We use networks where delays are unpredictable and packets get lost. We receive data from users or other systems that can be incorrect or unexpected. Our systems often run concurrently, creating highly complex synchronization problems.

Precisely because the real execution environment is complex, dynamic, and prone to failures, our abstractions are forced to simplify, make assumptions, and ignore parts of this complexity. They cannot encompass all possible nuances and edge cases of the real world. And as soon as reality deviates from the simplified model presented by the abstraction (e.g., the network runs slower than expected, or incorrect input arrives), the details the abstraction tried to hide come to the surface. This is why, in the real world, the “leakiness” of abstractions is not the exception, but the inevitable rule.

A prime example of “leaky” abstractions is virtualization technologies, such as virtual machines (VMs). Modern virtualization technologies like virtual machines (VMs), Docker containers, and the Windows Subsystem for Linux (WSL2) create powerful abstractions that simplify development, deployment, and system management. They provide the illusion of isolated, self-contained environments, as if applications or entire operating systems are running on separate hardware. However, these abstractions inevitably “leak,” exposing the complexity of the real world, design compromises, and technological limitations. To effectively use such systems, solve performance issues, or ensure security, specialists must deeply understand the implementation details and account for these “leaks,” rather than blindly trusting the claimed isolation.

Virtual Machines (VMs)

Virtual machines create an abstraction of hardware isolation, allowing operating systems to run in isolated environments as if on separate physical hardware. However, this abstraction “leaks” for several reasons related to the complexity of the technology and compromises in its implementation:

• Performance: The actual performance of a VM depends on the load on the host machine from other “neighbors” (the “noisy neighbor” effect), the host’s processor model, and hypervisor settings. The abstraction of “dedicated resources” breaks down due to shared hardware and compromises in managing shared resources, leading to unpredictable deviations from expected performance.
• Hardware features and vulnerabilities: Processor vulnerabilities like Spectre and Meltdown became a stark example of hardware details “leaking.” They exploit speculative execution and caching mechanisms — a result of compromises in CPU design for speed. An attack from one VM can, through side channels (e.g., analyzing cache access times), read data from the host or other VMs, shattering the fundamental abstraction of isolation. CPU microarchitectural details that the hypervisor should hide “leak” upwards, creating security threats and requiring complex patches at all levels.
• Network interaction: Network latency and throughput in a VM depend on the configuration of virtual switches and the host’s topology, differing from the physical network. This is a manifestation of the complexity of adding virtual network layers and the compromises in their implementation.
• Hypervisor complexity: The hypervisor itself is a complex software layer with its own bugs, overhead, and peculiarities affecting guest systems. This reflects the fundamental complexity of virtualization technology and the inevitable trade-offs between isolation and performance.

Because of these “leaks,” specialists cannot view VMs as perfectly isolated “black boxes.” Solving performance problems, ensuring security (especially after vulnerabilities like Spectre/Meltdown), or effective administration requires a deep understanding of how the hypervisor, processor, and network operate, as well as accounting for limitations and side effects caused by the imperfect abstraction.

Note: You can find some more background information on different virtualization technologies you can read here https://alex-ber.medium.com/dns-resolution-service-inside-docker-container-on-wsl2-072a24d873f6

Docker Containers

Another example of “leaky” abstractions involves more lightweight technologies like Docker containers. Docker containers on Linux create the illusion of an isolated process in its own environment, but their lightweight nature is achieved by using the host OS’s shared kernel. This shared kernel, a compromise for speed and efficiency, becomes the primary source of “leaks”:

• Kernel vulnerabilities: If the host kernel has a vulnerability, for example, allowing privilege escalation (like Dirty COW), a process in a container can exploit it to attack the host or other containers. This breaks the abstraction of isolation, showing that system security depends on the security of the shared kernel — a consequence of its complexity.
• Kernel dependency: Applications in a container might depend on a specific version of the host kernel or its modules, breaking the abstraction of a “self-contained environment” and exposing the complexity of the underlying system.
• Resource contention: Even with limitations via cgroups, intensive use of shared kernel resources (disk I/O, task scheduler) by one container can degrade the performance of its neighbors. The abstraction of a “guaranteed resource slice” leaks due to real contention and compromises between limitations and full isolation.

These “leaks” show that Docker containers do not provide perfect isolation. Their effective use and security require understanding Linux kernel workings, cgroup mechanisms, and the limitations of the shared architecture.

Windows Subsystem for Linux (WSL2)

Yet another example of “leaky” abstractions is the Windows Subsystem for Linux. WSL2 provides the abstraction of a full Linux environment integrated with Windows but uses a lightweight virtual machine. The “leaks” are related to the specifics of this VM and compromises made for integrating the two systems:

• Resource management: The vmmem process, responsible for the WSL2 VM, can hold a significant amount of host memory even if memory is freed within Linux, hindering Windows applications. This is a “leak” related to the complexity of VM resource management.
• File system performance: Accessing Windows files from WSL2 (via /mnt/) is significantly slower than working with files in the “native” Linux file system within the WSL2 virtual disk. This exposes the boundary between the systems that the abstraction tries to hide.
• Network interaction: The WSL2 virtual network adapter can cause peculiarities in application behavior or performance issues, a consequence of network virtualization complexity.
• Overhead: Running a full Linux kernel in a VM requires startup time and constant consumption of host resources, which “leaks” as information about the real costs of this abstraction.

These limitations show that WSL2, despite its convenience, is not a seamless integration. Specialists must consider the specifics of the VM, file systems, and network settings to optimize performance and troubleshoot issues.

Virtualization and containerization technologies like VMs, Docker, and WSL2 offer powerful abstractions that simplify system development and management. However, their “leakiness” — a consequence of the complexity of real systems and design compromises — manifests in performance issues, security vulnerabilities, and dependencies on underlying technologies. Effective use of these technologies requires specialists to have a deep understanding of their internal implementation and to account for the “leaks,” rather than blindly trusting the claimed isolation or simplicity. Ignoring these details inevitably leads to problems, whether it’s reduced performance, vulnerabilities, or unpredictable system behavior.

Why Does a Professional Need to “Look Deeper”?

If abstractions are meant to simplify life by hiding details, why should a professional spend time and effort studying what’s “under the hood”? The answer is simple: because abstractions, as we’ve established, “leak,” meaning reality is more complex than the interface promises. When a system doesn’t work as expected, it’s precisely the understanding of the underlying layers that becomes key to solving the problem.

• Diagnosis and Problem Solving: When a bug appears, an unexplained performance drop occurs, or simply strange, unexpected system behavior is observed, a superficial look at the level of the used abstraction is often insufficient. The cause might lie in those very “leaked” implementation details.

Is your graphical application becoming unresponsive when resizing the window or updating data on the screen? Perhaps the issue isn’t in your update logic, but in how the GUI framework you’re using performs redrawing and element layout calculations. A simple call to a refresh() method or changing a widget’s property might, under the hood, trigger a complex and resource-intensive process of recalculating geometry and redrawing for a large tree of interface components, even if the visible changes are minimal. Without understanding the internal layout and rendering mechanism hidden behind simple API methods, optimizing interface performance will be extremely difficult (leakage of the GUI framework abstraction). Or perhaps you’re reading a file byte by byte using a simple read function, unaware that under the hood this leads to numerous expensive system calls and disk operations (leakage of the file I/O abstraction).

Does the program throw a strange error not described in the API documentation? It’s likely an error from a lower level (network, file system, OS) that the abstraction couldn’t fully hide or handle correctly.

Is your asynchronous application incorrectly processing requests, pulling in someone else’s data or settings? The problem might lie in the incorrect use of context-local variables (e.g., ContextVar in Python). You might have expected the request context (user ID, interface language) to be automatically preserved in asynchronous tasks or callbacks, but due to specifics of context copying or incompatibility with a library, it gets lost or replaced by another context. The abstraction of “automatic context management” has leaked, requiring an understanding of the actual rules of its propagation and lifecycle.

Without understanding how the abstraction works internally and what details it tries (but doesn’t always succeed) to hide, diagnosing such problems turns into guesswork. Knowledge of the “internals” allows formulating correct hypotheses, using the right analysis tools (profilers, low-level debuggers, system monitors), and finding the root cause, which is often one or more abstraction layers below the one you are directly working with.

Furthermore, understanding what lies behind the interfaces of abstractions is critically important for making informed decisions when choosing tools, technologies, and architectural approaches. Development is a continuous process of choices, and the quality of these decisions directly depends on the depth of understanding of the available options.

Choosing a programming language, framework, database, library, or even a design pattern should not be based solely on a superficial description of their capabilities or marketing promises. A professional must be able to look deeper:

• Assessing real suitability: A tool might look perfectly suitable based on its API, but its internal implementation (e.g., algorithms used, concurrency model, data storage method) might make it inefficient or unsuitable for the specific performance, scalability, or reliability requirements of your project. Understanding the “internals” allows assessing how well the tool will actually cope with the task at hand.
• Understanding and weighing trade-offs: Every technology is a set of compromises. When choosing between SQL and NoSQL databases, different types of cloud storage, or various frameworks, you are choosing not just an API, but a specific set of architectural decisions with their strengths and weaknesses. Understanding these internal decisions (data consistency models, storage mechanisms, query processing methods) allows you to consciously choose the compromise that is most acceptable for your goals.
• Anticipating future problems and limitations: Knowing about potential “leaks” and limitations of an abstraction in advance allows anticipating possible bottlenecks, integration difficulties, or operational complexities in the future. For example, when choosing a specific network library, it’s useful to know how it handles errors or manages connections to avoid unpleasant surprises as load increases.
• Comparing alternatives: When faced with several similar tools offering comparable high-level functionality (e.g., two different message queues), the deciding factor often lies precisely in the differences in their internal architecture and implementation (delivery guarantees, persistence model, clustering capabilities). Understanding these differences allows choosing what best fits the project’s non-functional requirements.

Relying solely on the external interface of an abstraction when making technological decisions means risking choosing a tool that later proves to be suboptimal, difficult to maintain, or unable to handle the real load. Deep understanding allows making strategically correct choices based on how systems actually work.

Finally, a deep understanding of abstractions and their “leaks” is crucial for ensuring the security of developed systems. Vulnerabilities often arise precisely where reality diverges from the simplified model offered by the abstraction, or where an attacker can exploit “leaked” details for their benefit.

Security is not just about adding encryption or authentication at a high level; it must be built into all levels of the system, and this requires understanding potential weaknesses, often hidden behind convenient interfaces:

• Identifying and preventing attacks on lower levels: Many classic vulnerabilities, such as SQL Injection or Cross-Site Scripting (XSS), exploit precisely the “leaks” in abstractions. The data access layer abstracts database interaction, but if it improperly handles user input, an attacker can pass malicious SQL code that “leaks” through the abstraction and executes at the DBMS level. The UI generation system abstracts HTML markup creation, but without proper escaping of user input, a malicious script “leaks” onto the page and executes in the client’s browser. Understanding how the abstraction interacts with the lower level (DBMS, browser) allows anticipating such attacks and applying correct protection methods (e.g., parameterized queries, proper output escaping).
• Analyzing side channels and information leaks:”Leaks” can manifest not only as direct execution of malicious code but also as information leakage through side channels. Differences in system response times, details in error messages, changes in resource consumption — all these can reveal internal structure or data that the abstraction was supposed to hide. As the Spectre and Meltdown attacks showed, even processor microarchitectural features can become sources of critical leaks. Understanding these potential channels is necessary for designing systems resistant to such attacks.
• Correct security configuration: Setting security parameters often requires understanding how not only the abstraction itself works but also the system beneath it. For example, configuring firewall rules for containers requires understanding their network model and interaction with the host. Managing access rights in an application often depends on how the operating system or database implements access control at a lower level. Blind faith in the “security by default” provided by an abstraction can lead to serious configuration errors.
• Assessing the security of dependencies: Modern applications are built from numerous third-party libraries and frameworks — ready-made abstractions. Assessing their security is possible only by understanding not just their API, but also how they are implemented and what potential “leaks” or vulnerabilities they might contain.

Ignoring what happens “under the hood” of abstractions creates blind spots that attackers can exploit. A development professional must be able to analyze the system from a security perspective at different levels, anticipating how “leaks” can be used for attacks and taking measures to prevent or mitigate their consequences.

Note: While I was writing this story, I unexpectedly encountered DNS resolution issues. After some investigation, it turned out the problem was with the DNS resolution service inside a Docker container on WSL2. This came as a complete surprise to me, despite the fact that I had written a note about it myself https://alex-ber.medium.com/dns-resolution-service-inside-docker-container-on-wsl2-072a24d873f6

The Trap of Excessive Abstractions: When Simplification Complicates

Abstractions are necessary for managing complexity, but they are “leaky”, and it’s important for professionals to understand their internal workings. However, excessive use of abstractions can turn a tool for simplification into a source of problems. Paradoxically, too many layers of abstraction complicate the system, making it convoluted and inefficient.

Each new layer of abstraction simplifies interaction with the underlying level but adds indirection and potential “leaks”. When there are too many layers, the system turns into a complex “layer cake,” where simple operations require passing through numerous stages.

Reasons for Complexity and Inefficiency:

• Increased indirection: To perform an operation, a request passes through multiple layers, each doing its own work, transforming data, or calling the next layer. This makes understanding the execution flow and debugging difficult.
• Reduced performance: Each layer introduces overhead (function calls, checks, data transformations). Their accumulation, or the “abstraction penalty”, leads to noticeable slowdown compared to a direct approach.
• Increased resource consumption: Multi-layeredness requires more memory to store intermediate data and the state of each layer.
• Complexity of understanding and maintenance: Developers have to consider not only the top-level API but also the interactions of intermediate layers to work effectively with the system or fix errors. The overall picture becomes blurred.
• Hiding, not solving, problems: New layers are often created to mask the complexity or “leaks” of underlying levels, rather than fixing them. This is a consequence of either excessive theorizing or attempts to follow trendy architectural patterns. Such an approach only worsens the situation, adding inefficient and potentially problematic abstractions on top of an already complex foundation.
• Decreased Performance: The overhead of each layer adds up, increasing latency, reducing throughput, and increasing CPU and memory consumption. Simple operations slow down with no apparent benefit to the user.
• Debugging Complexity: Finding the source of an error becomes an ordeal. A problem originating at a lower level might manifest differently at the top level after passing through several processing stages. The debugger has to navigate all layers to find the root cause, which is complicated by loss of context and data transformation.
• Difficulty in Understanding: Multi-layeredness hinders the formation of a holistic view of the system. Developers need to understand not only the API but also the interaction of all layers to predict system behavior or assess the impact of changes. This increases cognitive load.
• Maintenance and Evolution Complexity: The above factors increase maintenance costs. Difficulty in understanding and debugging slows down changes and bug fixes. Modifying one layer can unexpectedly affect others, increasing the risk of new bugs. Implementing new features becomes laborious, and onboarding new team members becomes more challenging.

The pursuit of simplification through excessive abstractions, especially if they mask problems, leads to the creation of cumbersome, slow, and hard-to-understand systems. A simple task requires passing through a complex and opaque mechanism, and the system becomes fragile and expensive to maintain. Instead of simplification, the opposite is achieved: complexity, inefficiency, and increased costs.

An excellent example illustrating the trap of excessive abstractions is the comparison between the theoretical OSI (Open Systems Interconnection) network model and the practically dominant TCP/IP model.

OSI vs TCP/IP

The OSI model, developed by ISO, is a conceptual framework for network device interaction, divided into seven clearly defined layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application. The idea behind OSI was strict separation of responsibilities. Each layer was supposed to solve its narrow task, provide services to the layer above, and use services from the layer below, completely abstracting away their details. In theory, this promised maximum flexibility, interchangeability of protocols, and ease of understanding.

On the other hand, the TCP/IP model, which formed the basis of the modern Internet, has a more pragmatic structure, usually described with four layers: Link, Internet, Transport, and Application. This model was developed not so much as a theoretical scheme but as a real working set of protocols, and it combined some functions that were separate in OSI.

In practice, it turned out that the strict seven-layer structure of OSI contained layers whose functionality was often unnecessary or easily implemented at other levels. Specifically, the Session layer, responsible for managing dialogue between applications, and the Presentation layer, handling data format conversion and encryption, often proved redundant. In real systems built on TCP/IP, their functions are either embedded directly into the Application layer or handled by other mechanisms, such as TLS/SSL for encryption, which operates on top of the Transport layer.

Furthermore, implementing and standardizing protocols for all seven clearly separated OSI layers proved to be a complex and slow task. Although the OSI model did not gain widespread adoption in its pure form, one can assume that passing data through all seven layers, each with its own processing, could have led to greater overhead compared to the more “flat” TCP/IP model.

Ultimately, the simpler, more flexible, and earlier-developed TCP/IP model became the de facto standard for the Internet. The OSI model remained largely an academic concept and a reference model for studying networks. This example perfectly illustrates how a theoretically elegant division into multiple layers of abstraction can turn out to be impractical and redundant in the real world. Adding layers of abstraction does not always lead to improvement and can sometimes unnecessarily complicate the system.

Data Access Layer

Let’s consider another classic case where one might encounter the trap of excessive abstractions: choosing how to implement the DAL (Data Access Layer) in a typical layered application architecture.

Modern applications are often built on a layered principle for separation of concerns. Simplified, it looks like this:

• Presentation/Controllers Layer: Responsible for user interaction (web interface, API endpoints). Frameworks (e.g., Spring MVC, ASP.NET Core, FastAPI) are often used here to help process requests and responses.
• Service Layer: The main business logic of the application is concentrated here. This layer orchestrates the execution of operations, coordinates interaction with other systems, and makes decisions. Ideally, this layer should not depend on specific details of the user interface or data storage method; it operates on pure business entities and processes.
• Data Access Layer (DAL): Exclusively responsible for interacting with the data store (e.g., a relational database). Its task is to provide the service layer with a convenient interface for saving, retrieving, and modifying data, while hiding the specific details of SQL queries, connection management, etc.

It is precisely at the DAL level that the question arises: which abstraction should be chosen for interacting with the database? And here, one of the popular, but potentially “excessive”, options is a full-featured ORM.

What is an ORM and why is it needed if we consciously chose a relational DBMS? An ORM is one way to build a DAL. Its main goal is to bridge the conceptual gap (impedance mismatch) between the object-oriented model used in the application code (classes, objects, inheritance, relationships) and the relational model of the database (tables, rows, columns, foreign keys).

• Automated Mapping: ORMs handle the routine task of converting data from table rows to application objects and back. This can significantly reduce the amount of boilerplate code, especially for simple CRUD (Create, Read, Update, Delete) operations.
• Working with Objects: Allows business logic developers (at the service layer) to operate with familiar objects and their relationships, thinking less about writing SQL for each operation.
• Standardized Data Access: Provides a unified API for interacting with the database, which can simplify development and maintenance.
• Additional Features: ORMs often offer built-in mechanisms for transaction management, data caching, change tracking (Unit of Work), which can speed up the development of certain features.

The idea is that, having chosen the power and reliability of a relational DBMS, you get a tool that allows interacting with it in a more convenient, object-oriented way, especially for typical tasks.

However, this powerful abstraction comes at a cost, and its use can be an example of redundancy:

• Complexity of the ORM itself: Full-featured ORMs are complex systems with their own world model (sessions, object states, lazy/eager loading). Learning and correctly using an ORM can require significant effort, adding another complex layer to understand.
• The “leaky” abstraction problem: ORMs try to abstract away SQL, but often fail. ORMs try to abstract away SQL, but often fail. The classic “N+1 query” problem: An innocent-looking method call like author.getRelatedPosts() or simply accessing a collection of related objects (e.g., when iterating through an author’s posts) can, under the hood, lead to executing one SQL query to load the main object (author) and then N separate queries to load each related object (post), where N is the number of related objects. This is extremely inefficient and a common cause of performance problems hidden behind a convenient object interface. To solve performance problems (like N+1) or to write complex, optimized queries, the developer still needs to understand what SQL the ORM generates and often intervene in this process using ORM-specific query languages or writing native SQL. The abstraction has “leaked.”
• Opacity and “Magic”: The automatic behavior of ORMs makes debugging and profiling difficult. It’s hard to understand why a particular query is executed or why performance drops.
• Overhead: The complex mechanisms of ORMs introduce noticeable overhead in performance and memory consumption.
• Object-Relational Mismatch: ORMs don’t always smoothly handle the fundamental differences between object and relational models, especially when working with complex database schemas.

Instead of a full-scale ORM, the DAL can be implemented using more direct approaches that provide more control over SQL while still offering some conveniences. Using “helpers” to execute SQL: Instead of working directly with the low-level database API, one can use libraries or framework components that simplify routine tasks: connection management, query execution, exception handling, and basic mapping of results to objects. However, the SQL query itself is written by the developer.

• Example in Java: JdbcTemplate from the Spring Framework. It allows writing SQL as strings, passing parameters, and provides convenient ways to map result rows (ResultSet) to Java objects (e.g., using RowMapper). It eliminates the need to manually open/close connections and handle many JDBC exceptions, but control over the SQL query remains with the developer.
• Example in Python: SQLAlchemy Core. Unlike SQLAlchemy ORM, Core provides tools for executing “raw” SQL strings, passing parameters safely. It also helps manage connections and transactions but doesn’t attempt to completely hide the relational model or automatically map objects in complex ways.

These approaches offer a lower level of abstraction over the database compared to a full-featured ORM. They require the developer to write and understand SQL, but in return provide more control, transparency, and often better performance. The developer explicitly manages SQL, transactions, and mapping (albeit with some help from helpers), making the system’s behavior more predictable. This often turns out to be a good compromise between a complete lack of abstraction and the complex “magic” of an ORM.

An ORM is a powerful tool for building a DAL, but its use is a prime example of a trade-off. It can significantly speed up the development of simple CRUD operations. But for complex systems with high performance requirements or when working with non-trivial queries, a full-featured ORM can become an excessive abstraction, adding more complexity and problems than it solves. Choosing a “thinner” abstraction layer for the DAL, where SQL is managed more explicitly, might be a more pragmatic solution. It’s important to choose a tool whose level of abstraction and complexity matches the actual needs of the project, rather than blindly following trends or the initial promise of completely eliminating SQL.

Conway’s Law: How Team Structure Shapes System Design

Conway’s Law, formulated by Melvin Conway in 1967, states:

“Organizations which design systems are constrained to produce designs which are copies of their communication structures.”

This means that the architecture of software — modules, components, interfaces — inevitably reflects the organizational structure of the teams developing it. For example, if three teams work on a product, the system will likely consist of three subsystems with interfaces corresponding to the communication channels (or lack thereof) between these teams. Conway’s Law highlights how the social aspects of development influence the technical outcome.

At the heart of Conway’s Law lies the cost of communication — the time and energy expenditure required for interaction between people. In software development, this cost determines what the system will look like, especially at the boundaries between teams.

Inter-team communication is complex, expensive, and slow due to:

• Additional effort: Coordination requires meetings, material preparation, context explanation, agreement documentation, consuming time from both sides.
• Risk of misunderstanding: Differences in specialization, geography, or organizational structure increase the likelihood of errors in interpreting requirements or interfaces.
• Formal processes: Interaction requires creating tasks in trackers, writing API specifications, conducting reviews, adding bureaucracy and delays.
• Different priorities: Teams with different goals and KPIs make finding mutually beneficial solutions difficult.

Due to the high cost of communication, the system minimizes inter-team interaction. Interfaces between components developed by different teams become rigid, formal, and stable, even if technically suboptimal. This hinders optimizations requiring changes across multiple system parts, such as reworking data formats or moving logic between components. Such improvements are often postponed because coordination costs outweigh the benefits. Conway’s Law explains why technically feasible optimizations are often not implemented, as the system “freezes” in a structure reflecting organizational boundaries.

Within a single team or in the work of a single developer, communication is fast, informal, and cheap thanks to:

• Ease of coordination: Ideas are discussed instantly, without complex processes or meetings.
• Shared context: Team members share an understanding of tasks, codebase, and goals, reducing the risk of misunderstanding.
• Informality: Interaction doesn’t require strict protocols or bureaucracy.
• Fast feedback: Changes are tested and implemented without lengthy approvals.

Under such conditions, optimizations occur more easily and frequently. If a developer sees an opportunity to improve interaction between modules they are responsible for, or if a team decides to refactor, barriers are minimal. As Casey Muratori noted,

“low-cost areas get optimized assuming good specialists are involved, but high-cost areas do not.”

The cost of communication determines which parts of the system evolve and which remain “frozen.”

Technical boundaries in the system — modules, microservices, APIs — coincide with the organizational boundaries of teams. When two teams develop interacting components, an API emerges between them, becoming a formal contract. Due to the high cost of inter-team communication, such APIs are:

• Stable and rigid: Changes require complex negotiations, so APIs are made immutable, even if suboptimal.
• Reflect compromises: API design is the result of negotiation, not purely technical decisions.
• Minimize interaction: APIs are simplified to reduce dependency and communication frequency.

Thus, abstractions not only reflect the organizational structure at the time of creation but also fix it in the code. Even if the team structure changes, reworking established interfaces remains costly, preserving the “imprint” of the old organization. Technical abstractions become artifacts solidifying past organizational decisions.

Windows’ volume control

A striking example of Conway’s Law is the multitude of volume controls in Windows, especially in older versions. Users face confusion: the main volume control in the system tray, the mixer for applications, settings in the Control Panel, utilities from sound card manufacturers (Realtek, Creative), and controls within programs. This complexity is a consequence of Conway’s Law stretched over time:

• Different teams and eras: Sound in Windows was developed by different Microsoft teams (OS kernel, interfaces, applications) at different times, each adding its components and interfaces. The company’s communication structure changed, and new features were layered onto old ones.
• Third-party developers: Hardware manufacturers created their drivers and utilities independently of Microsoft, adding their abstractions on top of system ones.
• Application developers: Program creators added their own volume controls, interacting with complex system APIs that were themselves multi-layered due to historical legacy.
• Backward compatibility: Old mechanisms were not removed, but new ones were added alongside, intensifying the chaos.

The multitude of sliders is a physical manifestation of the boundaries between teams and eras, fixed in the interface. This shows how Conway’s Law, over time, creates complex and suboptimal systems, even if each team acted rationally.

Conway’s Law emphasizes that system architecture is not only a technical solution but also a product of organizational structure and communication cost. Understanding this helps to consciously design teams and processes to minimize barriers to optimization and create more effective systems.

Conway’s Law at the Micro-Level: Code Structure and Cognitive Load

The principle of communication cost influencing design manifests not only at the team level but also in code structure, especially in object-oriented programming. The choice of abstraction methods can create barriers to understanding and optimization, even if the code is written by a single developer or a small group. Each class can be viewed as a “team” with an internal mechanism (private methods and data) and an external interface (public methods, API). Ill-considered use of abstractions, such as deep inheritance hierarchies or excessive polymorphism, distributes related logic across files and classes, increasing the cognitive “communication cost” for the developer, complicating analysis and refactoring.

Let’s examine this using the example of calculating the area of geometric shapes in C++, illustrating the influence of code structure on optimization, as described in my article “Melvin Conway’s nightmare or The Only Unbreakable Law” (https://alex-ber.medium.com/melvin-conways-nightmare-or-the-only-unbreakable-law-297b2bbcf23b).

Note: You can find the actual C++ code in the link above.

• Polymorphic Approach: Uses dynamic polymorphism via virtual functions, ideal for open sets of types where frequent addition of new types is expected (e.g., plugins or user extensions). An abstract base class shape_base defines an interface with a pure virtual function Area() and a virtual destructor. Derived classes (square, rectangle, triangle, circle) implement Area() to calculate the area (e.g., Side * Side for a square, Pi32 * Radius * Radius for a circle). The TotalAreaVTBL() function calculates the total area using an array of shape_base pointers, employing dynamic dispatch (vtable). However, if the set of shapes is stable (as is often the case with basic geometric forms), the Area() calculation logic is fragmented across classes, increasing cognitive load. The developer needs to study the Area() implementation in each class, track the hierarchy, dynamic calls, and manage dynamic memory. This increases cognitive load, hindering the understanding of the overall logic and optimizations (like switching to GetAreaTableLookup()). Additionally, the overhead of vtable and allocation reduces performance.
• Tagged Union with Switch: Employs a tagged union (shape_union) with an enumeration shape_type (Shape_Square, Shape_Rectangle, Shape_Triangle,Shape_Circle) and a struct containing the type and data (e.g., square_data with a Side field, rectangle_data with Width and Height). The GetAreaSwitch() function uses a switch statement to select the area formula based on the shape type. The TotalAreaSwitch() function sums the areas by iterating over an array of shape_union. The code is simpler, localized in one function, requires no dynamic memory (array on the stack), and runs faster, avoiding virtual calls. The area calculation logic is gathered in one place, reducing cognitive load and facilitating analysis.
• Optimized Table-Lookup Approach: Uses a struct shape_data_opt with Type, Width, and Height fields (where Width is the side for a square, radius for a circle, base for a triangle; Height is unused for square and circle). The GetAreaTableLookup() function calculates the area by applying a coefficient from the CTable table (1.0 for square, 0.5 for triangle, Pi32 for circle), eliminating the switch branching. The TotalAreaTable() function sums the areas. The code is more compact, faster due to minimal overhead, and the logic is even more localized. The uniform data structure simplifies maintenance if new shapes fit the model.

Noticing the opportunity for optimization (e.g., transitioning from polymorphism to the table-lookup approach) is much harder in the first, fragmented version due to high cognitive load. If the code is more localized (“low communication cost”), the chance of seeing and implementing the optimization is significantly higher.

The difference between data modeling approaches, such as polymorphism and tagged unions, can be explained through category theory, where data types are represented as categorical sums and products.

Note:

* If you want to refresh you knowledge about basic C/C++ syntax (struct, union) you can read Categorical Sum and Product types using struct tagged union in C https://alex-ber.medium.com/implementation-of-categorical-sum-and-product-types-using-struct-tagged-union-in-c-c01cdbb13793

* If you want to see some theoretical insight on the difference polymorphism vs tagged union you can read Java’s record and sealed classes as categorical product and sum types parts I, II, III. It uses new Java features to demonstrate them.

• Dynamic Polymorphism: Uses an interface (shape_base with Area()). Method calls are resolved via vtable. The system is “open” to new types, but adding operations (e.g., perimeter()) requires changing the interface and classes, making the system “closed” for operations but “open” for types.
• Tagged Union / Sum Type: Implements a categorical sum: Shape = Square + Rectangle + Triangle + Circle. A shape_union value is one of the variants, and the data (e.g., rectangle_data = Width * Height) is a categorical product. Processing via switch or pattern matching makes adding operations (e.g., GetPerimeterSwitch()) easy, but adding a new type (e.g., Ellipse) requires modifying all switch statements, making the system “closed” for types but “open” for operations.

This dilemma, known as the Expression Problem, highlights the difficulty of creating a system that is flexible for both types and operations.

Choosing between polymorphism and tagged unions depends on the context:

• Open Type Sets: For scenarios with frequent type additions (e.g., plugins), polymorphism via interfaces (shape_base) is more convenient, as it’s open to new types but closed to new operations. This provides extensibility without code modification, making polymorphism a powerful tool for flexibility.
• Closed Type Sets: For stable sets, like shapes (Shape), tagged unions (shape_union, shape_data_opt) are preferable, as they are closed to new types but open to new operations. The logic for operations like GetAreaSwitch() or GetAreaTableLookup() is localized, reducing cognitive load and simplifying optimizations, such as transitioning from switch to CTable. Modern languages (Java, Kotlin, Swift, Scala, Rust, F#) support this through sealed classes and pattern matching, ensuring type safety and logic localization.

Applying polymorphism to stable type sets, as in TotalAreaVTBL(), fragments the Area() logic across classes (square, rectangle, triangle, circle). The developer must switch between files, track the hierarchy, and dynamic calls, increasing cognitive “communication cost” and hindering optimizations like switching to GetAreaTableLookup(). This acts like an “internal” Conway’s Law: logic fragmentation creates barriers similar to inter-team ones. In approaches with shape_union (GetAreaSwitch()) and shape_data_opt (GetAreaTableLookup()), the logic is gathered in one function, reducing the load. For instance, in GetAreaSwitch(), area formulas are immediately visible, and in GetAreaTableLookup(), there’s a uniform model with CTable. This simplifies analysis and optimization, akin to the low communication cost in a cohesive team. If the code is created by a single developer or a cohesive team, the low “communication cost” increases the chances of improvements, as noted by Casey Muratori: “assuming good specialists are involved,” optimizations are inevitable.

The choice between polymorphism (shape_base, Area()) and tagged unions (shape_union, shape_data_opt) impacts performance, code locality, and cognitive load. Polymorphism is valuable for open type sets but can lead to logic fragmentation for stable sets, acting as an “internal” Conway’s Law. Tagged unions are effective for closed sets, minimizing barriers and simplifying optimizations, especially with sealed classes and pattern matching. Understanding categorical sums, products, and the Expression Problem helps choose an approach that balances flexibility, simplicity, and optimizability.

Navigating Levels and Finding Balance: Key Skills of a Professional

So, we’ve seen that abstractions are a double-edged sword. They are necessary for managing complexity, but they can “leak”, their excess complicates the system, and organizational structure leaves its mark on the design. How can a professional work effectively under these conditions? It requires developing specific skills, chief among them being mental flexibility.

A professional cannot afford to be either a blind follower of abstractions, ignoring the reality “under the hood,” or a perpetual skeptic, rejecting the convenience of abstraction for fear of “leaks”. Mastery lies in mental flexibility — the ability to effectively use abstractions to solve problems at a high level, while being ready and able to “descend” to the implementation level when necessary.

This means:

• Conscious use of abstractions: Understanding why a particular abstraction is used, what problems it solves, and what details it hides. Using its benefits to speed up development and simplify code where appropriate.
• Readiness to “dive deeper”: Not treating an abstraction as an impenetrable “black box”. Being prepared, when problems arise (bugs, poor performance, unexpected behavior) or when deep optimization is needed, to lift the “lid” and understand the details of the underlying layer.
• Ability to connect levels: Understanding how actions at a high level of abstraction translate into operations at lower levels, and how “leaks” from lower levels affect the behavior of the upper level. Seeing cause-and-effect relationships across layers.
• Context switching: The ability to quickly shift focus between the overall system architecture and the specific implementation details of a single component or algorithm.

Essentially, this is the skill of “zooming” — the ability to see both the forest and the individual trees, and even the leaves on them, switching between these scales as needed. Without such flexibility, a developer either gets stuck at a high level, unable to solve deep-seated problems, or drowns in implementation details, losing sight of the big picture and the benefits of abstraction.

Simply being able to “descend” to lower levels is not enough. An effective professional must also understand and anticipate the limitations they will inevitably face. These limitations come in two main types:

Technical Limitations (“Leaks”): These are the very “holes” in abstractions we discussed. Understanding the Law of Leaky Abstractions is not just stating a fact, but actively knowing potential problems in the tools and technologies used. A professional must:

• Know the typical “leaks” for their technology stack (e.g., how garbage collection works in their language, what performance issues the ORM or framework might have, how network protocols affect latency).
• Anticipate where an abstraction might “break” (edge cases, high loads).
• Understand which implementation details of a specific abstraction might affect security, performance, or reliability.
• Be able to find information about the internal workings of used abstractions when necessary.

This knowledge allows not only faster diagnosis of problems when they arise but also designing more robust systems initially, considering potential “leaks” and choosing abstractions with known and acceptable limitations.

Organizational and Communication Limitations (Conway’s Law): Beyond purely technical aspects, a professional must be aware of the influence of team structure and communication costs on the project. This includes:

• Understanding how the current or past team structure has already influenced the system design (why modules are divided this way, why this API is so complex).
• Anticipating which changes or optimizations will be easy (within one team, low communication cost) and which will be difficult or impossible (requiring coordination between teams, high communication cost).
• Being able to account for these organizational constraints when planning work, estimating timelines, and choosing architectural solutions. Sometimes a technically “perfect” solution is unrealizable due to organizational barriers.
• The ability to communicate effectively across these organizational boundaries when necessary, understanding the associated costs and risks.

Understanding both technical “leaks” and organizational limitations allows a professional to operate not in a vacuum of ideal abstractions, but in the real world with all its complexities and compromises. This helps make more informed decisions, set realistic goals, and create systems that not only work but can also be maintained and evolved under existing conditions.

The Art of Choice: Finding the Right Level and Number of Abstractions

Software development is largely a process of creating and using abstractions. And here, there is no universally “right” answer to the question of what level of abstraction to choose or how many layers to use. It is always a context-dependent decision, requiring the professional to have a fine sense of balance and the ability to weigh various factors.

Finding the right level of abstraction:

• Not too low: Using excessively low-level abstractions (or none at all) where good high-level solutions are available leads to writing a lot of boilerplate code, slows down development, and increases the likelihood of errors. One must know how to use ready-made tools that solve standard problems.
• Not too high: Using an overly high-level, “magical” abstraction where fine control, high performance, or handling specific edge cases is required can lead to the problems described earlier (“leaks,” inefficiency, debugging complexity).

The professional’s task is to choose, for each specific task or system component, the level of abstraction that provides the best compromise between development speed, ease of use, flexibility, performance, and control.

Finding the right number of abstractions:

• Avoid unnecessary complexity: As discussed in the section on redundancy, adding too many layers of abstraction can make a system convoluted and inefficient. Each new layer must bring tangible benefits that outweigh the complexity and overhead it adds.
• Don’t fear simplicity: Sometimes the simplest solution, with the minimum number of layers, is the best one, even if it doesn’t seem as “architecturally elegant”.
• Consider Conway’s Law: When designing layers and interfaces, it’s important to understand how they relate to the organizational structure. Sometimes it’s better to choose a slightly less “clean” architecture that better aligns with the communication possibilities between teams than to create a system that is perfect on paper but impossible to develop effectively.

This is an “art” because it often requires experience, intuition, and understanding of the specific project context (requirements, team, deadlines, expected system evolution). The right choice of abstractions is about finding the golden mean between the chaos of low-level details and the labyrinth of excessive layers, between development speed and long-term maintainability, between the technical ideal and organizational reality.

Awareness of Trade-offs: No Silver Bullet

Perhaps the most important conclusion from the entire discussion of abstractions is that any abstraction is a compromise. There are no perfect abstractions that are simultaneously simple, powerful, fast, flexible, absolutely reliable, and completely hide all details.

When we create or choose an abstraction, we inevitably trade one thing for another:

• We might gain ease of use at the cost of reduced performance or loss of control over details. (Example: Using a standard function to read an entire file into a single string or list of lines is very simple, but can be extremely inefficient in terms of memory and speed for very large files compared to stream reading and processing data in chunks).
• Flexibility and power might require a more complex interface or increased learning time. (Example: Graphics libraries like OpenGL or Vulkan provide immense power and control over the GPU but have very complex APIs and steep learning curves compared to high-level graphics engines).
• High performance can often only be achieved at the cost of more complex code and abandoning convenient abstractions. (Example: low-level optimizations).
• Complete hiding of details can lead to unexpected “leaks” in edge cases.
• Following the team structure (Conway’s Law) can lead to a technically suboptimal design.

A professional must be clearly aware of these trade-offs for every abstraction used or created. This means:

• Not believing in “silver bullets”: Being skeptical of any tools or approaches that promise to solve all problems without any drawbacks.
• Actively looking for the downside: When evaluating a new technology or pattern, asking the question: “What’s the catch? What compromises were made here? What do I lose by using this?”
• Weighing pros and cons in context: Understanding which compromise is acceptable for the specific project and its requirements. What is a drawback for one system (e.g., high memory consumption when reading an entire file) might be completely irrelevant for another that only works with small configuration files, where code simplicity is more important than memory savings.
• Being ready to pay the price: If you choose an abstraction for simplicity, be prepared for potential performance issues. If you choose control and performance, be prepared for more complex code.

Awareness of the inevitability of compromises dispels illusions and allows for informed, pragmatic decisions. Instead of searching for a non-existent ideal, the professional focuses on choosing the best available compromise for the specific situation, understanding both the benefits and potential drawbacks of their choice. This is the foundation of a mature engineering approach.

Risks of Imbalance: The Price of Ignoring Complexity

We’ve discussed the importance of understanding “leaks”, avoiding excessive abstractions, and considering organizational factors. But what happens if a professional or team ignores these aspects? Imbalance — whether it’s blind faith in abstractions, thoughtless layering, or ignoring the influence of team structure — leads to serious risks and the accumulation of technical debt.

The consequences of such imbalance manifest in many ways:

• Systems become fragile, unreliable, slow, and convoluted. Ignoring “leaks” leads to unpredictable behavior and failures in real-world conditions. An excess of layers accumulates overhead, reducing performance, while a rigid structure dictated by Conway’s Law blocks optimizations. Excessive layering, especially to mask problems, turns the architecture into a tangled labyrinth, difficult to understand and debug. Instead of reliable, fast, and understandable systems, we get their opposites.
• The team loses the ability to solve complex, non-standard tasks and perform deep optimization. When a system is convoluted or built on poorly understood abstractions, solving problems beyond routine becomes extremely difficult. Lack of knowledge about underlying layers or the inability to refactor due to architectural or organizational barriers prevents achieving required performance. The team becomes dependent on the “magic” of abstractions and helpless when it fails.
• People problems arise: hiring and onboarding become difficult. A high entry barrier to a complex project slows down the adaptation of newcomers. Knowledge transfer becomes complicated, and documentation often doesn’t reflect reality. Experienced developers, tired of fighting the system’s complexity and fragility, may leave the project, and attracting new strong specialists becomes harder. This creates a negative spiral, further exacerbating maintenance and development problems.

All these problems sum up to the accumulation of technical debt. Ignoring “leaks” is like taking out a loan with future “interest” in the form of complicated maintenance. An excess of layers is a high-interest loan where complexity grows exponentially. A rigid structure dictated by Conway’s Law is the inability to “refinance” the debt through architectural improvements. Eventually, the “loan payments” (effort spent on maintenance and workarounds) can become overwhelming, threatening the future of the entire project. The inability to find balance in working with abstractions is a direct path to creating systems buried under the weight of their own technical debt.

Conclusion: The Dual Nature of Abstractions and Professional Mastery

We have journeyed from the basics of abstraction to the complexities of its real-world application, examining “leaky” abstractions, the traps of redundancy, and the influence of organizational structure. In conclusion, it is crucial to re-emphasize the dual nature of this powerful tool.

Abstractions are an indispensable tool for managing complexity. Without them, modern software, and indeed the entire technological world, would be impossible. The ability to hide details, highlight essentials, break down large systems into manageable parts, and reuse ready-made solutions is what allows us to create incredibly complex systems and drive progress forward. From programming languages and operating systems to libraries, frameworks, and architectural patterns — abstractions lie at the foundation of all our engineering activities. They save our mental effort, allow specialization, and enable building anew on top of the old. This is their undeniable value.

But extremes are dangerous for the professional: blind faith, thoughtless layering, and ignoring reality.

Despite all their benefits, abstractions are not a panacea, and their incorrect use is fraught with serious problems. As we have found, for the professional creating and maintaining complex systems, several extremes are equally dangerous:

• Blind faith in the perfection of abstractions: Perceiving abstractions as absolutely hermetic “black boxes”, ignoring the Law of Leaky Abstractions, means building fragile systems and being unprepared to solve real problems arising from “leaks”.
• Thoughtless piling up of layers: Getting carried away with creating ever newer levels of abstraction without a clear understanding of their necessity and cost means risking the creation of a convoluted, slow, and hard-to-maintain system, where the very means of combating complexity becomes its source.
• Ignoring the influence of organizational structure (Conway’s Law): Designing a system in isolation from the reality of communication flows and team structure means creating an architecture that may be suboptimal from the start and extremely difficult to evolve and optimize in the future due to the high cost of interaction across organizational barriers.

Falling into any of these traps leads to the accumulation of technical debt, reduced product quality, and increased difficulty for the team.

True professional mastery in working with abstractions lies not in blindly following dogmas (be it “abstract everything” or “no abstractions”), but in finding a wise balance. It manifests in the ability to:

• Understand boundaries: Clearly recognize both the technical limitations and “leaks” of the abstractions used, as well as the organizational and communication barriers (Conway’s Law) affecting the project.
• See opportunities for optimization: Be able to look “under the hood” of abstractions and through organizational structures to find real bottlenecks and opportunities for improving performance or architecture, even if they are not obvious on the surface.
• Make conscious choices: Flexibly select the right level and number of abstractions for each task, weighing all trade-offs and avoiding both unnecessary complexity and excessive simplification.
• Interact with the actual implementation: Confidently dive into details when necessary — whether it’s diagnosing database locking issues, or initiating the refactoring of a shared component with other teams — without being confined by the provided abstractions.

The mastery of a developer or architect is the art of navigating the complex landscapes of abstractions and organizational structures. It is the ability to harness the power of abstraction without becoming its slave, and to work with the complexity of the real world without drowning in it. It is precisely this balance that allows the creation of systems that are not only elegant on paper but also reliable, efficient, and capable of evolving over the long term.