This post will be my perspective on software design, a compilation of discussions and examples of different principles that we associate with software design.
Keep in mind that this is not meant to be a crash course in software design. If you are interested in such content, I recommend reading System Design by Vasa, Philosophy Of Software Design By John Ousterhout, and The Pragmatic Programmer by David Thomas, Andrew Hunt.
Why do we need software design? Poorly designed systems can lead to catastrophic outcomes for the business. Such as unmaintainable or hard to debug, not extendable, and not scalable systems.
First, let's set a definition of software design. Software design is the process of managing tradeoffs when architecting a system. A system in this definition can be anything from a single process/thread program to a multi-cluster multi-region distributed system.
Simplicity
Simplicity is the most important, easiest to understand, and most challenging to master principle of design. You may have heard of the KISS (keep it simple, stupid) principle. But what does it mean for a system to be "simple".
I define a simple system as a system that can be understood by anyone with minimal domain knowledge. For example, a simple accounting system can be understood by any accountant or any developer with enough accounting background. Keep in mind that simple does not mean feature lacking. On the contrary, simplicity allows for extensibility, as we will discuss in the next section.
As you can infer from the definition, designing a simple system requires understanding the business domain. I previously discussed how understanding the problem helps in driving the solution. Similarly, understanding the business domain helps in shaping your system's design. I would go as far as to say you can infer how much a programmer knows in their domain when reading their code. Vagueness in understanding leads to poorly written code resulting in complexity.
Also, you should avoid any fancy design patterns or tools. If you use a design pattern just because it sounds cool or is commonly used in a certain community, you will introduce unnecessary complexity to the system that could've been avoided. I will share one example inspired by a real-world codebase – this is just one out of many.
class Vehicle:
# Moves the vehicle for the specified duration and returns how far it moved in floats.
def move(self, minutes) -> float:
...
class Car(Vehicle):
def drive(self, mode, seconds) -> float:
self.move(seconds / 60)
...
class Motorcycle(Vehicle):
...
def get_distance_travelled(vehicle: Vehicle) -> float:
distance_travelled = 0
if isinstance(vehicle, Car):
distance_travelled = vehicle.drive("sport", 120)
elif isinstance(vehicle, Motorcycle):
distance_travelled = vehicle.move(2)
return distance_travelled
In the previous example, we had a base-type Vehicle that has been extended twice. And it has a method move that takes the duration in minutes and then returns a float of how far this vehicle moved in this duration of time. However, when the new type of Vehicle, Car, was made, the developer found that its concept of moving and its requirements were different than the default move. So, they made a new drive method that took a different combination of parameters.
Given this difference in the interface, the developer had to distinguish between vehicles in the call site and treat them differently, breaking the original premise of inheritance.
Because we do not have enough business context for this example, I won't propose a solution, but this shows how this confused understanding of polymorphism vs. inheritance made this inconsistent design. We would've been better off if we did not make a base class for our vehicles to inherent and relied on duck-typing or interfaces depending on the language.
As you can see, simplicity is hard. It takes a lot of discipline and thought to get anywhere near a simple design.
Scalability
Scalability can be divided into two subcategories extensibility and stability, or functional scalability and load scalability.
By extensibility, I mean the capacity to extend the system by adding new features or altering existing ones. When designing any program, small and large, always think of what features can be added. You do not necessarily have to think in-depth of the implementation details as much as having a vision of how to expand your feature set. Having this in the back of your mind helps you design for expansion.
On the other hand, stability may be the easier of the two. There are so many tutorials on improving the performance and stability of the specific framework or tool you are using. Things like eager loading vs. lazy loading in web apps, caching, JIT compiling, to name a few. However, I summarize all of these tutorials this way: do not do what you do not have to do. Most optimizations revolve around the idea of minimizing resources usage. If you load too much data, you are using too much memory and storage (and maybe network). If you loop too many times, you are using too much compute, and so on.
Keep in mind that the key to scalability is simplicity. If the system is too complex for you to comprehend, it's really hard to know what to optimize.
Design Process
Design is an iterative process. You come up with a cool idea to comply with particular constraints in your requirements, only to realize that it introduces more limitations or does not entirely cover the whole case. You then go back to the drawing board and either refine the proposed solution or start over.
One great way to reduce the time spent trying to implement a solution and later having to redo it is by having design workshops. Having concerned parties of a particular component meet and discuss a solution. Talk about what benefits a solution would provide or the problem it would solve, and most importantly, what constraints it introduces to the system. Also, you can invite people who do not directly deal with the system to have a set of fresh eyes to give a different perspective (this has proven to be very helpful in multiple situations).
Design should not be done in a vacuum. Once you get your first couple of thoughts on paper, try presenting them to others. Trying to explain an idea is the best way to scrutinize it. For example, one time, I had to design a schema for a routing system for shipments. I knew that I needed to know where the shipments are coming from, going to where, on what service type, by whom, and for how much. So, I decided to have each one of these questions solved by a database table.
One table defined origin cities. Another table linked the origin cities to the destination cities for specific service level named destinations. The last table had linked the previous table with couriers and their prices. After sketching this design on paper, I showed it to two fellow engineers on the team. We discussed how this proposed design would be used in code and debated some of the field names. For example:
destination.origin_city vs. destination.origin
destination.cost
...
Given that this destination table was the core piece of our system, we focused on its usage. After a couple of examples, both of the engineers found this confusing. Then we realized that all of this could be reduced to the concept of a route. One table has origin, destination, service_level, and cost fields. And we did not need to debate the naming:
route.origin_city
route.destination_city
route.cost
...
That would not have been achieved without sharing and discussing the design with others, especially potential maintainers of the system.
The first design is probably the worst. So do not get attached to your first idea. Instead, keep challenging it until you verify that it achieves your design goals.
Abstractions
As Butler Lampson says, "All problems in computer science can be solved by another level of indirection." Abstraction (indirection) is a fundamental idea in software design. It helps you define what you are actually trying to solve. Most software products are abstractions of some sort, whether it is an intermediate piece of software used by other software (e.g., APIs) or an end application used by the end-user. We are always trying to abstract something.
So, this makes sense, but how do we tell if a problem should be solved by abstraction? And how do we define the scope of a given abstraction problem?
These two questions can be answered by answering two other questions: how common is this problem? And how wide is the range of use-cases? Let's take, for example, the problem of sending HTTP requests to external web services. How often do we send HTTP requests from our app? If we only have one instance of sending an http request in our app, we should not bother to abstract it. However, if we do have multiple instances, we should look into variance in usage. For example, are these all the same exact request used in various places? Are these requests sent to the same API provider? Are these all the same HTTP method? And so on.
Once we understand the broadness of our use cases, we can start designing our abstraction. Let's say I am building a Twitter client. In this case, I will be interacting heavily with the Twitter API. So, I will make an abstract approach to communicating with it. Look at the following function signature:
def call_twitter(endpoint: str, method: str, params: dict = None) -> dict
The call_twitter takes an endpoint which is the path of the API I want to interact with. I should not include the host information or API version that I have abstracted away from the user. The method is simply the HTTP method that can be passed in lower or upper case. Lastly, the optional params and the return value are dicts given that the Twitter API always takes and returns payload wrapped in an envelope as described here.
In a call site, I would do something like:
tweet = call_twitter("/tweets/2244994945", "GET")
Which would return something like:
{
"data": {
"author_id": "2244994945",
"created_at": "2020-06-24T16:28:14.000Z",
"id": "1275828087666679809",
"lang": "en",
"possibly_sensitive": False,
"source": "Twitter Web App",
"text": "Learn how to create a sentiment score for your Tweets with Microsoft Azure, Python, and Twitter Developer Labs recent search functionality.\nhttps://t.co/IKM3zo6ngu"
}
}
You may also want to explore whether or not to unwrap the response and return thedata field directly as you do not want to end up with tweet["data"]all other the place.
With that being said, some "attempts" at abstractions may yield harmful results. These attempts can be divided into two categories. The first one is immature abstractions. That is when you try to implement an abstraction for the use case that is not well understood. These usually occur in abstractions of business logic. For example, let's say we are trying to implement a tax calculator. The naive way of doing it is to simply make a function that takes the price and multiply it with the rate.
TAX_RATE = 0.15
def calculate_tax(price: Decimal) -> Decimal:
return price * TAX_RATE
The problem with this design is that it does not account for products with different tax rates and exempted products. If we then needed to support those, we will have to take one of two approaches. The first approach is to handle the exception in the call site, which means the callers need to be smarter. The other approach is to change the function signature to either take the tax rate or the product category and decide the rate internally.
def calculate_tax(price: Decimal, category: str) -> Decimal:
return price * get_tax_rate(category)
However, this will require that we change all callers to pass this parameter. Note this may seem like a trivial example, but these small things can cause many cascading problems when you're not accounting for this missing value. My rule of thumb here is If you do not understand the domain, do not try to abstract it.
The second category of harmful abstractions is fake abstractions sometimes called empty or pass through abstractions. Look at the following example adapted from real world production codebase:
def update_product_quantities(order, operator):
for order_product in order.products:
if operator == "+":
order_product.product.quantity += order_product.quantity
elif operator == "-":
order_product.product.quantity -= order_product.quantity
Then someone came and decided to abstract this function by doing the following:
def deduct_products_quantities(order):
update_product_quantities(order, "-")
def return_products_quantities(order):
update_product_quantities(order, "+")
As you can see, the two functions only have one line of code each. They do more harm than good, and they are not really abstracting anything. I need to memorize more terminology deduct and return, which are not even proper antonyms. I am assuming that the intention of the developer who made these function to hide the cognitive complexity of the operator parameter. However, they ended up introducing a similar amount of cognitive load when they introduced the new terminology.
Closing Thoughts
Software design is not at all about memorizing common architectures and sprinkling fancy design patterns but instead figuring out what to do and when. Furthermore, it is about doing the least amount of work that would give you the most mileage (remember, simple and scalable).
I may have spent most of this post on abstractions, but I believe that our work as developers is all about abstractions. We apply all the previous principles when we implement abstractions. An abstraction needs to be simple and scalable, which can be achieved by having a good design process.