Software Projects and Diseconomies of Scales

Economies of Scales

In IT industry, we often hear the term economy of scale, especially those who work closely with Cloud Computing. In fact, one of the main selling arguments of cloud computing is the fact that it leverages economy of scale. But what is exactly economy of scale? In economics it is:

Economies of scale refer to reduced costs per unit that arise from increased total output of a product. For example, a larger factory will produce power hand tools at a lower unit price, and a larger medical system will reduce cost per medical procedure.

http://www.investopedia.com

In similar way, in Cloud computing, it means large cloud vendors reduce the cost per produced cloud computing unit (e.g. processing unit, storage unit, … etc.), which allows them effectively to compete with other traditional on-premise computing methods. The result as we all witness, cloud computing is taking the IT world by a storm!

Diseconomies of Scales

Now we understand what Economy of Scale is; the question is, what is Diseconomies of Scales? At first glance, it seems to be the exact opposite of economies of scales – smaller organization suffer increase in the cost of produced units. No, not that straight forward. Well, let’s first start with the economical definition then we break it down:

Diseconomies of scale happen when a company or business grows so large that the costs per unit increase. It takes place when economies of scale no longer function for a firm. With this principle, rather than experiencing continued decreasing costs and increasing output, a firm sees an increase in marginal costs when output is increased.

http://www.investopedia.com

Here is a graph shows where Diseconomies of Scales happen:

As you can see, Diseconomies of Scale occur after Economies of Scale for the same organization. I do not want to delve into the reasons behind it in general sense (mostly economical explanation), but I do recommend reading this wonderful article for details.

What I am after here is, a phenomena appears in Software Development projects. This phenomena is strongly linked to Diseconomies of Scale.

Diseconomies of Scale in Software Development Projects:

I will put it bluntly, although the total team productivity increases in general, the individual productivity level drops; as the team size increases. This is also know as Ringelmann effect, which states the following:

The Ringelmann effect is the tendency for individual members of a group to become increasingly less productive as the size of their group increases.

Ringelmann study ascribed the phenomena to the social loafing and loss of coordination. However, it was long before it was realized that loss of coordination is deeper that what was initially thought. As the team size grows, more coordination is required to compensate the loss thereof. Where am I a going with this?

Well, to compensate loss of coordination, typically means to enforce, encourage, or allow more coordination. This is what Steve McConnel refers to as communication overhead, which eats up into the individuals productivity. The real problem is this does not happen linearly. If you have two team members, then there is only one communication channel. If you have three team members, the communication channels suddenly increase to three; because A needs to coordinate with B and C (that is two communication channels), and C  and B need another channel to coordinate with one another. How about four team members? Six! Yes, six communication channels. Well, there is a formula for that:

No. of communication channels = n * (n – 1) / 2

where n is number of the team members (team size)

But what is coordination here? What kind of work that people waste their time on, instead of doing the actual work? Well, it is actually an actual work. Coordination may include dividing tasks, resolving dependencies, teaching one another, side (important) conversations, meeting delayed because one out of ten was a bit late, and so on, so forth. It is basically what keeps the team in sync. Without it, you’ll end up with silos produce non-functioning or crippled software, which if it works today, has no chance to be extended, or sustained tomorrow. Hence, we agree the coordination is part of the job, and it is not fluffing around.

Now, the formula above brings us back to Brook’s law

Adding manpower to a late software project makes it later.

Actually Brook’s law is not only because of the communication overhead, but certainly one of the reasons a team hit Brook’s law point is the saturated communication channels. That said, adding more staff to a team, increases the team overall productivity despite the drop in the individual productivity. However, there is a point, there is a tipping point where adding more staff to a team drops the individuals productivity such that even the overall team productivity stops improving, and then worse yet; it starts to dropping. That is where Brooks’ Law meets Ringelmann effect, and Diseconomies of Scales dynamics kick in. That is point Q on the graph above my dear.

Every time, as a project manager, or a team manager/lead you want to make a decision about either committing timeline, staffing for project, revising late project; think about these variables, and add them to your decision equation. Think about Brook’s Law, Ringelmann effect, Econonomies and Diseconomies of Scale, and last but not least; think about the communication channels before you add manpower to a late project. Because if you understand and follow these laws and facts, you’ll be equipped with great tools that will help you to make the right decision. In the long run, your team and your customer will thank you for it.

 

 

Fast estimates, Slow delivery

I just came back from my holiday in Koh Samui and Koh Phangan islands. I spent few days enjoying the sunshine, blue skies, and mesmerizing beaches. It was also an opportunity for me to plough through Daniel Kahneman‘s masterpiece, Thinking, Fast and Slow. I cannot count the new fascinating ideas and thoughts I learned from the book. Perhaps not the most interesting one, but certainly the idea of The Planning Fallacy is something all of us, IT people, can relate to very well. Simply put:

phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed.

The Planning Fallacy

The author explains further, the term Planning Fallacy describes plans an forecasts that:

1. are unrealistically close to best-case scenarios
2. could be improved by consulting the statistics of similar cases.

In Software development world, we all suffer the planning fallacy. In all my years of IT, I have not encountered a single case, where the project was delivered ahead of time. It does happen, but no so often, that it never happened to me at least. Developers are optimistic by nature, which adds more fuel to the planning fallacy. One of the reasons I believe the developers fall into this fallacy, is that they like to code, and they enjoy doing challenging and new things. Unbeknownst to them, they tend to underestimate the unknown, and underestimate again to be allowed to do the unknown. I can now think of more than one case, where I unconsciously played down the new challenges ahead of tasks, because they seemed interesting, and I wanted to do these tasks. I did not want my manager to change the direction, and think of easier, and perhaps boring solution – which effectively would have negative impact on the outcome as well. Now, this kind of reason is a result of a bias thinking, it is very personal, and one cannot always have full control over it, without much of discipline. At the same time, I can imagine, someone else, playing up some challenges, because s/he does not want to do the task, and hopes the manager will redirect the work to something perhaps easier.

The Fallacy Remedy

Ignoring the reasons above, and suppose we all aim for unbiased estimates. Granted, there will be still numerous challenges. But at least, how could you eliminate the planning fallacy we mentioned. First, and foremost, ignore your intuition. That is right. I repeat it again, ignore your intuition. There is not enough space to discuss how our intuitions are more often than not, misleading. But if you want to know why, I would advise you to read the aforementioned book. For now, the gist of the reason is that intuition is often wrong in highly irregular environment. Even if the task is similar, it is in the nature of software development that the environment is highly volatile, e.g.: The same function works fine in one project, but cannot handle the load of another project. The result, you cannot simply copy and paste. Once you eliminate your intuition, use the scientific method to give the estimate, as the author proposes:

1. Identify an appropriate reference class  (in our case, similar project, or task)
   • We used this in our software development team. Started by all developers sat together in a brainstorming session. We thought of every possible abstract task. We tried to make them as generic as possible (we did that before I stumbled upon the planning fallacy concept, but certainly we were suffering from it). One example, create one REST API, is a class of tasks. Another example, create one database table is another class of tasks, and so on.
2. Obtain the statistics of the reference class.
   • At the beginning, there are no statistics, so we assigned best-guess estimates to each class of tasks. Overtime, we adjusted that to reflect more realistic estimates.
3. Use specific information about the case to adjust the baseline prediction.
   • Suppose, you are creating one REST API, you know on average how long will take to complete. At the same time, you know, you need to add pagination, or you need to add some caching technique. Here, you adjust the estimate accordingly.

This might seem a lot of work. But think about this: You have a reference list created once for all classes of tasks you know. You keep updating it with the new kinds of tasks (this becomes less frequent by time). All developers, easily refer to it, and adjust their estimates. If you are following Scrum, and all estimates avoided the planning fallacy, you can imagine, how much your team estimates will be more accurate and realistic. The benefits of this method outweigh the cumbersome of the small additional work. In addition, it takes a lot of estimating burden away off the shoulders of the developers. Instead, they focus their effort on estimating the adjustments to the special cases.

Summary

Software estimate is one of the hardest topics in software engineering. There are myriad books, essays, and research papers on the topic. They cover various sides of it, but many miss the planning fallacy, and advocate the expert judgment. Unfortunately, the expert judgment is not always correct, and could lead to disasters. In psychology, nothing beats the systematic way of estimating and making decision. This is particularly true in highly irregular environment such as software development.

Dockerize Java Play App

I heard about Docker a lot. In fact, all the microservices I developed in the past year are deployed using Docker. Having said that, my knowledge about Docker was somewhat limited to knowing what is Docker, and not how to use it – that was the task of one of our developers who is passionate about setting up things, unlike me who is passionate more about writing the code after things are already set up. Few days ago, I decided to fiddle with Docker myself and dip my toes into the practicalities of it. Here I am sharing my experience.

Generate and run Docker image

First, to generate your Docker image from your own Play App, navigate to the App root directory, and simply run the following in your command line, to generate Dockerfile and Docker assets:

$ sbt docker:publishLocal

It will generate docker directory under: target/

Navigate to target/docker/stage/, then by using the below command, you can generate your first Docker image (will be generated under target/docker/stage/ )

$ docker build -t flying-thots-docker-image .

To run your Docker image in the background, use the following command:

$ docker run -p 9000:9000 -d flying-thots-docker-image

To stop it, I usually get the container if from using:

$ docker ps

Then:

$ docker stop 

Using different conf file

Let us be honest, the above is very trivial, and it is mentioned tons of times on the web. The challenges start when we come down to real life scenarios. One of the main scenarios in Play Apps, is to run on production with different configuration file – typically we call it prod.conf . With typical Play App, that is very easy by passing the path to the configurations file. For example:

$ flying-thots -Dconfig.file=prod/flying-thots/conf/prod.conf -Dplay.http.secret.key=hidemeonproduction

When you run it inside Docker, however, it becomes a bit tricky. The reason is that Docker image has its own file system, and inherently its own file paths. In order for you to access files on the host system from the container, you need to map these files. I only discovered this by accident while struggling with my App not running. This post gave me the hint Play Framework 2.4 docker image run error. I tried with that, but initially did not work. Then I found this post How to access a directory in hosts machine from inside a docker container?, which suggests that you need to use the absolute file path in your mapping for both sides – host and container.

A little bit more on mapping

The syntax of mapping configuration file on the host file system to Docker is like this:

-v param1:param2

where

param1: /absolute/path/to/file/on/host

param2: /absolute/path/to/file/on/dokcer

Docker absolute file in my case starts from (root/target/docker/stage/). Hence, the path starts from /opt/…

Note that you could map directory or a file. However, be careful. If you map directory to existing one, it will override it. For example, I mapped conf directory from the host to /opt/docker/conf directory on the container. The conf directory on the host consisted only of prod.conf file, while the one on the container consisted of many other files. When I ran my image, it displayed many error messages related to these files being missing. Apparently, docker replaced conf dir completely with the conf dir from the host. Eventually, I decide to just map the prod.conf file only. However, I gave it a unique name so it does not overrid an existing file with the same name

Ultimately, this what worked for me:

$ docker run -p 9000:9000 -v /home/flyingthots/deployments/conf/application.conf:/opt/docker/conf/outside.conf flying-thots-docker-image -Dconfig.file=/opt/docker/conf/outside.conf -Dplay.http.secret.key="hidemeonproduction"

This blog post, goes through other options for getting application configuration into Docker container.

Disclaimer: The featured image is taken from devout.io

SCRUM for microservices

Scrum is one of main two agile frameworks along with Kanban. It is inherently adjustable to the team’s needs and requirements. There are tons and tons of articles on the web describe different approaches of Scrum. Most of these approaches more or less follow the Scrum guide . I have been working with Scrum for a many years with different teams and for different organizations. One of these organizations was relatively small with development team of 4 to 5 co-located developers who at any given time, work on the same project/topic. Another organization was massive. It had tens of development teams, each of which consists of tens of developers, who work from 3 different continents across the globe. You can imagine the starke difference between the two setups. It does not need much contemplating to realize they require different approaches of Scrum. The real work is how different and what is different.

In the recent years, the trend of adopting microservices has soared rapidly. The architectural pattern is forcing the organizations to adapt their teams structures to be more supportive of microservices. This is very expected behaviour if we consider Conway’s Law

“organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”

What does this mean for Scrum team involved in developing microservices? It basically means you need to adjust your team, and cannot keep it the same way it was when it used to develop monolethic system. The law is empirically proved. In microservices, usually one or two developers only are involved in developing one microservice. The microservice is self-contained, independent – or this is how it is supposed to be. This seems at the first glimpse conflicting with what the traditional scrum teams used to be. However, one should not overlook one of the main principles of scrum development team: self-organizing.

We recently faced the delimma of running a traditoinal scrum team, while trying to develop microservices. Most of the microservices required one of two developers only to work on. However, when we conduct our planning and estimation sessions, the entire team was involved in estimating stories, although most of them had no clue what the stories were about. The frustruation grew even further when the estimation gaps between the members were scarely variant, because each of them was working on different topic. This entailed long arguments, and the planning session turned to be tormenting for everyone. It took us a while before we figure out that the traditional scrum according to the book is not going to work for us. We decided to change the mindset. What was previously sacred and fixed, is now open for change. We held long discussions with everyone (developers, scrum masters, product owners, solutions architects). It took us a couple of months before we came up with a totally different approach, that respects the spirit of scrum, but bow to our team needs. The main points in the new scrum teams are as follows:

• Each microservice has a service owner (SO) and a back up service owner (BSO). Only those two developers are responsible for estimating their own stories.
• SO & BSO hold the deepest knowledge of their services, but other developers are also exposed to these services when needed. see below
• Before the sprint planning session, the Product Owner (PO) communicate with the respective SO and BSO for each service to get their estimates based on a new approach we put in place as a replacement for Planning Poker (which we ditched completely – I will write about this in a separate blog post). The SO and BSO agree on the SP for each story.
• During the planning session, each developer reports the capacity in time (not story points).
• Depending on the team velocity (e.g.: 200 hours the can complete 50 SP, then the velocity is 4), the scrum master will start assigning the stories to the development team.
• The stories are assigned to either SO or BSO first. However, if their capacity is fully utlizied, then the it will be assigned to different developer – this is how we foster knowledge sharing organically.