SOA Lessons: Don’t Put All Your Eggs in one Basket

When we think of the term “don’t put all of your eggs in one basket” in terms of IT, we more often think of not relying on a single vendor solution for everything. This is certainly a valid way to look at the term, but in SOA it also means we have to look at services and not just layers.

To put this in perspective, I am currently working on an external API solution for a large E-Comm company. In the current External API, which was released before I moved into the group, all of the services are found in a single “wrapper” project. On the plus side, it makes it easier to configure the services, as all of the configuration is found in a single web.config file. There are many negatives, however, which I will detail in this post.

Single Point of Failure

This is the largest negative on production. Recently, the solution was released and we discovered a vendor was overloading one of the services. The issue was discovered during deployment and manifested itself as failures in a wide variety of services. The reason for the systemic failure is the worker process for the failing service was the same worker process for all of the services. When the process was brought down, all suffered.

Fortunately, we knew which service was live and most likely to be the culprit, so we were able to alleviate the problem. If this was not the case, we would have had to pour through logs to determine the offender, which could have taken hours with the current state of affairs. That is unacceptable.

If you take this scenario a bit further and assume a coding error that kills the worker process rather than a load issue, imagine the troubleshooting. Especially if the exceptional case causing the failure was intermittent and due to the lack of a patch on the server in question.

One good reason to separate out the service endpoints is so you can move the services into their own process space. This is not mandatory in normal working conditions, but if there is an issue, and instrumentation and monitoring is not catching the culprit, separating the service into its own process accomplishes two things:

  1. Helps identify the point of failure
  2. Protects other services from failure

Both of the above are worthwhile reasons to separate each service into its own process space as a rule, and then making exceptions based on needs.

Single Point of Deployment

This negative is similar to the last negative, but focuses more on what it takes to fix a service that is in error. If all of the services exist in a single project, then all must be deployed at the same time, even if only one service has changes.

I think anyone reading this can see why this is a negative, but it is more insidious than moving pieces you should not have to move. Every time a software project is deployed, there is a potential for error. There are a variety of reasons why this is so, but we all have experienced a deployment that created a buggy condition. Often times it was code we did not even change where the bug shows up and often due to circumstances that have nothing to do with the code.

If you deploy a service that has no updates, and cause bugs due to mistakes in deployment, you have done a great disservice. Worse, the disservice was completely avoidable if you had simply segregated out the service so it did not require deployment with the other services.

Cross Contamination

When numerous service endpoints are added to a single WCF project, they will have different contracts, but often end up with the same binding rules and behaviors. This is normal and to be expected. Eventually, however, one service will be found to require more time to complete its work or a large payload (either request or response or both?) and edits will have to be made to the configuration file.

Changes that are specific to the contract and service endpoint are unlikely to have consequences outside of the service in question. But since at least parts of the configuration are shared, changes can impact the “sharing” services in a negative way. And, since we deploy at the same time, but often only test the service being updated, these negative consequences are often discovered by our consumers rather than our test team.

The Point

The point of all this is we can easily avoid the negatives explained in this post by making sure every UI project (in this case every WCF service) has its own project. Sharing a single solution is not an issue, although you will also want to make solutions for the individual services, as well. NOTE: Solutions are points of organization and a single project can exist in any number of solutions.

Separating out the services has a small negative of forcing you to make changes to multiple projects for “universal” binding and behavioral standards, but this negative is outweighed by the negatives of placing all of the services in a single project.

Peace and Grace,

Twitter: @gbworld

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: