Living in an Agile Biased World


I started today to write down some notes on the Wrox book Professional Test Driven Development with C#. I know James Bender fairly well and have met McWherter. But as I read I hit on a pet peeve of mine: arguing via bias (emotion versus logic). I thought about reviewing the book including this information, but it would not be fair, as the book should rise or fall on the merits of the material presented and not whether the authors have a strong bias towards Agile.

Let me back up a bit. I have been on the Internet since the late 80s. At that time I had to shell in via a BBS (Bulletin Board System) and use UNIX commands to navigate. Back then, arguing without a sound logical basis led to responses from the peanut gallery, especially if your faux pas was an ad hominem attack. If I go back farther in my history, I took a few logic classes (high school and college). The point is I had a background that acknowledged an argument is only as good as the reasoning behind it. A marketer would probably disagree. Smile

One common type of logical fallacy is the straw man argument. In the straw man, you misrepresent your opponents argument and then attack it with reasoning. Since you have set up “their” position, you can easily attack it. And that is what the author’s of the book do with the Waterfall development methodology.

The Straw Man

From page 5 of the book:

The concept behind waterfall was that every software project, whose average time span was about two years, should have every phase, from inception to delivery planned from the start.

So far, the definition is mostly technically correct. But it suggests that planning is done in a bubble. Read further:

This necessitated a long period a the beginning for requirements gathering. After all the requirements were gathered, they were “thrown over the wall” to the architects. The architects took the requirements and designed the system that would be built down to the smallest detail. When they completed this task, they threw the design over the wall to developers, who built the system.

Now this is where things start to break down. Even in the worst waterfall shop I have worked in, nobody threw things over the wall. In fact, architecture is almost always treated as a collaborative process, sometimes more effectively than others. In addition, even in larger organizations that had architecture groups, I have never seen architects isolated from developers. Maybe it happens in some organizations, but years of experience show me bad architecture is far more common than the ivory tower picture presented in the book (the words ivory tower architects are used on page 6).

Other things misrepresented:

  1. Waterfall testing is performed by manually running scripts (page 5) – Not since the early 80s in most shops
  2. All requirements are gathered early on and cemented (page 5) – While waterfall is more strict, there is a change management process when done correctly.
  3. The work must be estimated up front – This is technically correct, but in a good waterfall shop, estimates are estimates, not firm commitments

And here is the one that set me off:

Unlike waterfall, which seeks to control and constrain the realities of software development, agile methodologies embrace them.

Now this comment is just plain silly. Not only is it not supported by the text of the book, I am shocked that nobody called the author on this one. “Constrain the realities”? Really?

I was going to stop here, but chapter 2 has a comment that really kicked me in the gluteus maximum (page 24-25).

Imagine for a moment you are building a house. For most of us, a house is the most expensive thing we will ever buy. the opportunity to have a house built to your desires and specifications is something to be excited about. Imagine, though, that when you leave the architect’s office for the last time, you will not be allow to see the house’s progress or make changes to the blueprints or design until the house is completed. Imagine that the first time you get to see the house is the day you move in. How well do you think the house would reflect what you actually wanted? It may reflect what’s on the blueprints, but there’s a big difference between a drawing and a physical structure. Amazingly, this is how many companies still develop software.

I have to throw the bullshit flag on this one. While there are companies that attempt to develop in a box, without any feedback, they generally fall in the category “out of business”. Even in the strictest agile shops, we group functionality, have POCs, demos, and means of making changes to reflect changing business needs, goals and drivers. You don’t meet with the architect and then find what was done until the walkthrough. It may happen somewhere, but it does not happen often.

Okay, it sounds like I am picking on the authors, but what I am really on is the idea of misrepresenting something, whether it be a hacked conspiracy theory about aliens and 9/11 or a development methodology. Personally, I am an Agile junky to an extent, and a strong advocate of Test Driven Development (TDD), although I tend to move towards a more Behavior Driven Development (BDD) style. More on this in a bit.

Methodologies

What is waterfall?

In the waterfall methodology, you have a list of requirements you need to get completed. You also have a list of items the developers are currently working on. The full scope of the work and the current list are consolidated using a project plan, generally represented as a Gannt chart. As a particular requirement is completed, it is tested. When development is completed, the software is stabilized to eliminate bugs and released.

Okay, so what is Agile?

In the Agile methodology, you have a list of requirements you need to get completed. You also have a list of items the developers are currently working on. The full scope of the work and the current list are consolidated comparing the backlog to work done, generally represented as a burndown chart. As a particular requirement is completed, it is tested. When the development is completed, the software is stabilized to eliminate bugs and released.

Wow, they sound pretty much the same? Yes, at the high level. What are the differences?

  1. In waterfall, you generally try to get the FIRST PASS of the system defined through requirements before you start work. Once this first view is agreed on, you will need an approved change request with the new requirement. In Agile, you simply add the new requirement to the stack.
  2. In waterfall, you have gates at the end of a phase. This does not mean you cannot return to a previous phase and start forward again, or even send some of the group back into a previous stage. It does mean, when done correctly, that you get sign off at the end of a phase that the work is done, prior to moving full force into the next phase. In Agile, when done correctly, you iterate until you either clear all of the requirements, or (the bonus for the properly done Agile) until business states “can we release it now” and you go into stabilization.
  3. In Agile, done correctly, one aspect of development (time or features) is favored. In most cases (except Feature Driven Development), the key driver is time.Time is sliced up into small chunks (generally 2 to 3 weeks) and each piece of work is completed to at least beta quality and demoed. This is artificially accomplished in most waterfall shops, as well.

The Good, the Bad and the Ugly

There is good development and bad development. This is a truism whether you are using a waterfall methodology or an Agile one. The assumption in the book, and with many other developers who have embraced Agile, is waterfall is bad and Agile is good. Take, for example, this quote on page X:

[Agile Methods] are also not about chaos or “cowboy coding”. In fact, agile methodologies require a larger degree of discipline to administer correctly.

In many shops I encounter, Agile is chosen to facilitate a few cowboys who don’t want to plan or write documentation. I am glad the discipline is mentioned here, but it is more often missing in shops that claim to be Agile. I have heard the following in Agile shops:

  • Agile is about writing requirements during development – Then why are all of your TBIs (Team Backlog Items) one liners with empty task definitions.
  • My tests and code are my requirements documents – This only works if you are writing the correct tests and code.
  • If you have to comment, you are not doing Agile – Okay, I agree that comments in code often explain what the code does, rather than why, making them useless. And, I agree the discipline of refactoring to eliminate the comment is a great practice. Unfortunately, most of the cowboys I have encountered that dis comments fail to refactor enough to eliminate the need for a comment, even though they do not write a comment.
  • We don’t need documentation with Agile – No, we let the next group guess what our intent was and have people hack our libraries in strange ways to fit their requirements, leading to new defects that could have been avoided by documenting rather than forcing an exploration operation.

So Agile does not have a lock on good practices any more than waterfall has a lock on bad practices. Agile is another paradigm, not an evolution or revolution in development. Once you have that down, you start to realize you can take either method and make good software, or you can mix and match the best of each world. You just need everyone on board. So what is good, bad and ugly?

Bad waterfall is serial in nature,with no parallelism in the pieces. Good waterfall allows for overlap, and is fully collaborative, but signs off the phases in order to ensure things are not missed. Ugly waterfall is not only serial, but extremely rigid.

Bad agile is a group of mavericks pulling code with little response back to the business until the demo. Good agile is a collaborative effort that includes coders, business and testers in the team. Ugly Agile ends up with all code written unplanned in a stream of consciousness, often ending up with an overly complex solution that simply does not work.

The important thing to remember is you need all of the following to have a successful system built:

  • An understanding of what is being built before it is built. This can be up front (ala waterfall) or as work is pulled off the stack (ala Agile), but more likely you have a hybrid of the two. Even in the strongest Agile shop, the business guy who came up with the requirement probably has a good idea what it should look like and that has to be conveyed to be effective, whether it is on paper, in a meeting, on a phone call or sitting next to you (best in Agile).

The Real World

In the real world, you find the following:

  • Sans a strong case, you are stuck with a set of resources. This is a budgetary constraint.
  • Whether you are Agile or Waterfall, business has an idea when they want to release and it often a stake in the sand whether you piece out the work or do it all “big bang”.
  • In all systems, there is change during the project. Agile, on paper, has a better method of handling this change, but waterfall has a better method of planning and documenting the change.

In the real world, I find political fiefdoms whether the shop is waterfall or Agile. Waterfall allows a fiefdom to exist through rigid sign off procedures, so the one with the sign off pen takes control. Agile allows fiefdoms to exist by avoiding all rigidness and allowing the cowboys to take control. Both are dangerous towards getting development completed.

Which is Better?

This is like asking “which is better, steak or ice cream?” The best depends on the mood and tastes of the people going out to eat. If everyone has had lunch or it is a really hot day, ice cream will win out almost every time. But if you have not had dinner, a steak is a wonderful choice. And, if you play the dinner “date” right, you can have both steak and ice cream.

In some shops, you need the discipline of waterfall to get things done. This is especially true when an outside “agency” is doing work and it must fall within certain constraints (generally budgetary). I am not stating Agile does not work here, but keep going until we run out of money and we will pray it works as a project is a fear with Agile when this type of situation is encountered.

What I personally find is most effective is a mixture of the two. You need a certain amount of discipline and sign off to have a successful project, so pulling the gates from waterfall makes sense. During development, you need the flexibility to change, however, so running iterative cycles is a great option. You need to plan before developing, so forcing some type of up front documentation (ala waterfall) is a good thing (At minimum, you need a good definition of the use case with criteria of what acceptable means for the feature). But you also need representatives from business, development and test in the same team so progress is not impeded (ala Agile).

TDD Is Agile?

This is another misnomer I often hear. While TDD got it start in Agile shops, I have seen it effectively used in a variety of environments, including some that are extremely waterfall in nature. The same is true of many other Agile techniques, like pair programming. When we start mixing the techniques and the methodology, it becomes much easier to claim one methodology is superior, especially if we ignore techniques that are more likely employed by the opposing methodology.

The book often mixes the Agile technology of Test Driven Development (TDD) with the methodology, which helps build the straw man argument argument against waterfall. Unless we assume it is impossible to unit test in an upfront manner with waterfall, we cannot use TDD against the waterfall methodology.

Summary

Neither Agile or waterfall is a superior methodology in all cases. When done well, with the right business culture, either can be extremely effective. Done badly, both can be detrimental. Bad waterfall generally gets bogged down in process, delaying work that needs to be done. Bad Agile leads to unplanned and undocumented work, which requires tons of rework to get to a stable release.

In my experience, I find Agile done bad to be more dangerous, as overly rigid process generally has less impact on quality and maintainability. But I am still more inclined to push Agile as the primary methodology, as I find good Agile much more likely to produce good results.

(Not quite true, as I find using the gates after you pull the trigger to release, adds control to the chaos that might otherwise ensue).

Peace and Grace,
Greg

Twitter: @gbworld

FTC Nails Pomegranate Juicemaker for False Claims That Don’t Appear False


I found this one interesting. From the beginning of the press release … er … article (could not be a press release, right?):

“Administrative Law Judge D. Michael Chappell ruled on May 17, 2012 that some POM Wonderful ads are deceptive. The company’s ads claim that POM Wonderful 100% Pomegranate Juice and POMx supplements can ‘treat, prevent, or reduce the risk of heart disease, prostate cancer, and erectile dysfunction.’ ”

So the background is POM made a claim that their products can treat, prevent or reduce the risk of heart disease prostate cancer and erectile dysfunction. The FTC sued them for deceptive advertising and the standard they needed to uphold their claims was “competent and reliable scientific evidence”. Sans this, the complaint would be upheld and the FTC would win.

POM has conducted 10 clinical trials on their pomegranate juices and supplements (http://clinicaltrials.gov/ct2/results?term=POM+Wonderful).  Of course, these might be bogus, since they used hack institutions like John Hopkins, UCLA, University of Michigan, etc. And additional studies by the National Cancer Institute, MD Anderson, Sloan Kettring, etc. are also obviously slanted and cannot be used as proof (even though these institutions also study the seems to have a great case here.

If I move to peer-reviewed journals, there are only tens of thousands of hits, with thousands for heart disease and prostate cancer (along with other cancers) and quite a few hundreds on erectile dysfunction. Obviously, there is not enough evidence, cause an effective food or supplement would have millions of studies. After all, Zytiga, a new FDA approved drug has a bit over 700 scholarly articles (roughly the same as pomegranate and erectile dysfunction) and roughly 2500 if the word abiraterone, the chemical name for Zytiga (significantly less than the number of hits for Pomegranate and Prostate cancer, but who’s counting).

Zytiga is the obvious safe choice for discriminating patients, as the only major complications are hypertension (high blood pressure), fluid retention, weight gain, adrenocortical insufficiency (problems with adrenal glands) and hepatotoxicity (kills your liver). The non-safe choice, drinking pomegranate juice, is far worse, as drinking it in higher amounts might cause weight gain, according to various sites. Weight gain is far worse than heart attack in my book.

Here are a few links to journals with research on pomegranate (hack journals , of course):

A Google scholar search reveals more than 51,000 hits for pomegranate. If you go to more specific searches on pomegranate reveal more than 4,600 hits for cardiovascular, more than 3,000 for pomegranate and prostate cancer, and more than 700 hits for erectile dysfunction. Of course, since many, if not most, of these are peer reviewed journals and studies by government agencies, so they MUST be biased and unreliable (bad science like all peer-reviewed journals).

Let’s dig a bit deeper into our government, okay? Certainly they have something to say about how bad Pomegranate juice is for you. I mean, they couldn’t be saying something like pomegranate juice helps with medical conditions, right?

From the National Cancer Institute: 

“A study of 13 pomegranate compounds showed some were able to slow the growth and spread of prostate cancer cells and to cause cell death. Higher doses were found to be more effective.”

“Three types of prostate cancer cell lines were treated with either pomegranate extract, pomegranate juice, or two of their bioactive compounds. ALL (emphasis mine) pomegranate treatments were shown to increase cell death and decrease the spread of cancer cells, with higher doses found to be more effective.”

“Other studies in cancer cell lines found that the anticancer activity of pomegranate included effects on certain enzymes and pathways involved in cancer, such as the insulin-like growth factor (IGF) system.”

But this is the National Cancer Institute (NCI) and not the Federal Trade Commission (FTC) and agencies should only trust their own research, even if they don’t research medicines or food. Unless, of course, the research they are trusting is about claims on medications, then they should trust all research, right?

So who is Administrative Law Judge D. Michael Chappell? He MUST be an independent legal authority with no ties to either the FTC or POM, right? If not, he might be biased in one direction or another, which would seem almost unfair to opposing side. Well, a bit of investigation reveals him to be a completely unbiased employee of the FTC, so I am rolling with him all the way … just like the media that published the FTC press release as an authoritative source of information. Hey, if it is press released, it has to be right? And there is no bias against farmers, food, etc. in the government, right?

If it seems like I am a bit underwhelmed by our government, you hit the nail on the head. With the revolving door between various agencies and the industries they watchdog, and the history of going after food while giving “medicine” a relatively free ride (could be due to many federal employees being former {drug, food, ?} company executives/researchers/etc.), I am a bit leery when I see the FTC going after a food for false practices (at least they did not state “water can prevent dehydration” was a false claim, as the European Union did last year), so they have that going for them.

As an aside, here is a particular egregious revolving door case:

In order for the FDA to determine if Monsanto’s growth hormones were safe or not, Monsanto was required to submit a scientific report on that topic. Margaret Miller, one of Monsanto’s researchers put the report together. Shortly before the report submission, Miller left Monsanto and was hired by the FDA. Her first job for the FDA was to determine whether or not to approve the report she wrote for Monsanto. In short, Monsanto approved its own report. Assisting Miller was another former Monsanto researcher, Susan Sechen. Deciding whether or not rBGH-derived milk should be labeled fell under the jurisdiction of another FDA official, Michael Taylor, who previously worked as a lawyer for Monsanto.

Really? She writes a paper to get recombinant growth hormone approved for use in cattle and then gets to approve the research? Obviously no conflict of interest there, right? And I want the fox to guard my henhouse and should have a bridge to sell you later on today. Bleh!

Am I stating you should drink lots of pomegranate juice? Certainly not! Am I stating you should not take drugs if you have heart disease, prostate cancer or erectile dysfunction? Not at all! I am stating that POM’s “false claims” appear to have plenty of evidence behind them. And I would rather drink pomegranate juice than take a load of drugs (one for the cancer, another for the hypertension caused by the cancer drug, and another for the erectile dysfunction caused by the previous 2 drugs, and others to halt the liver damage, etc.).

And, yes, I am being a sarcastic snit at this moment.

Peace and Grace,
Greg

Twitter: @gbworld

FaceBook Marketing: Giving Businesses Free Advertising by Selling Your Reputation


I just recently got an email from a charity asking me to vote for funding. I am for the charity, so I went to this page to vote. I then found out the “details”.

In order to vote, I have to like Cultivate, a company I have never done business with. Until I log into FaceBook and like them, I cannot vote. And, once I do sign in, they have the right to state I voted on my page. Essentially, this is free advertising to all of my friends, every day, for a company I know nothing about.

The masses see this and think this company wants to help out. And they think they are giving the company a chance to help out their charity. But, in reality, they are giving the company free views every day on the largest social media site in the world.

Why the beef Greg? Well, if I can use an analogy … In the United States, you cannot hold a sweepstakes that forces you to buy. The reason for this law is it essentially ties a game of chance to purchase, making it easy to game the system so the winning entry stays within your inner circle. Now, companies certainly make it harder to enter without buying, but anyone can enter for free if it is a game of chance.

Now forcing someone to like you to is not illegal, but it holds the same principal. In order to play, you have to pay. Admittedly, the cost is very low (“give me some free social marketing”), but there is still a cost. And once they have access to your page, they have the option of “reminding” you and your friends daily, posting that you have voted, etc.

I would not be disgusted if Cultivate encouraged liking them or encouraged you to post that you had voted. This is good social marketing. But to force you to like them to even vote sends me the message “unless you give me some free advertising on your reputation, I won’t let you help YOUR charity”. The converse message is “No, I won’t help YOUR charity unless you buy me off with free marketing”. That is slimy.

I won’t do it. And every time I am hit with someone trying to get me to vote, I am going to respond negatively on social media to ensure they are aware why I am against this. I don’t want to support a business I don’t use simply to get a few dollars thrown at my charity. It makes me feel like a prostitute… and it should make you feel that way too. And, I would love it if you would tell these companies the same thing.

If you want to participate, you can start with Cultivate Wines (https://www.facebook.com/cultivatewines or Twitter @cultivatewines), as it is the one I saw. Assume they are ignorant of the sliminess of the tactic and how it goes against proper Netiquette. In other words, let them know, but be nice.

Peace and Grace,
Greg

Twitter: @gbworld

Best Practices: Organizing Development Projects


Today I am sorting through some problems with a check in (not mine, but helping determine root cause). I now have things down to a single error:

image

So, I know my problem is some source file is missing from the drive, but not from the project. This means an errant check in, so I am on track. The problem is I cannot find the source in Visual Studio very easily because the project has been organized into folders and this particular project does not fit the organization scheme.

So, this led me to write this entry about organization of Visual Studio projects. I have a few rules I follow:

  1. Be consistent
  2. Have the file system match the project organization (optional)
  3. When you rename something in a file (class, project name), be sure to rename the files
    This includes class files, names of projects, etc.

You can live with just one of these, but doing both gives you a great leg up when it comes to maintenance. Now, I know we don’t generally think much about maintenance while we are building, but maintenance ultimately costs an business more time and money than spending a bit of time to make sure your file organized in a uniform and consistent manner.

NOTE: As an aside, I did find the test by searching for {COMPANY}.AppFramework.Services.Providers.ESTests. I found that the project was actually named {COMPANY}.AppFramework.Services.Providers.IntegrationTests. I will cover this in point #3. Below is the first line of the AssemblyInfo.cs file in the project {COMPANY}.AppFramework.Services.Providers.IntegrationTests:

[assembly: AssemblyTitle("{COMPANY}.AppFramework.Services.Providers.ESTests")]

.csharpcode, .csharpcode pre
{
font-size: small;
color: black;
font-family: consolas, “Courier New”, courier, monospace;
background-color: #ffffff;
/*white-space: pre;*/
}
.csharpcode pre { margin: 0em; }
.csharpcode .rem { color: #008000; }
.csharpcode .kwrd { color: #0000ff; }
.csharpcode .str { color: #006080; }
.csharpcode .op { color: #0000c0; }
.csharpcode .preproc { color: #cc6633; }
.csharpcode .asp { background-color: #ffff00; }
.csharpcode .html { color: #800000; }
.csharpcode .attr { color: #ff0000; }
.csharpcode .alt
{
background-color: #f4f4f4;
width: 100%;
margin: 0em;
}
.csharpcode .lnum { color: #606060; }

1. Be Consistent

This one is a bit difficult to give a concrete set of rules for, as the nature of consistency really depends on the way your organize your projects. When I am building a project, I tend to organize into buckets based on functionality. The most common buckets (which correspond to folders in my Visual Studio solution) are:

  • Core – The actual business logic that performs the behavior for the solution
  • Domain – Models and exceptions used across many/all projects in the solution
  • Experience – Projects to gain access to the functionality in the solution (User interface projects, service projects, etc.)
  • Framework – Common functionality
  • Persist – Functionality to persist state between sessions
  • Test – Tests for the various projects

But this type of organization can be less effective when you further break down a very large project, as I am working on now. So, to the problem at hand. I think I am looking for an Enterprise Settings test project, which is part of the service model for the project I am working on, but it is not found in the solution at the correct spot. In section #3, I cover this a bit more, as was already mentioned. Here is the Solution Explorer:

image

2. Have the file System Match the Solution Structure (optional)

This particular advice need not be applied if you keep the file structure flat, meaning all of the projects are folders underneath the solution folder (default behavior). As soon as you start to organize the solution physically, you have to make sure the structure in the solution explorer matches, however. If you don’t it makes files very hard to find.

Since I organize by “type” of project (Core, Domain, Experience, Framework, Persist, Test, etc.), I find it just as easy to create the projects in the proper location. Now, I am quite religious about this when I am creating projects for user’s groups and conferences, as it makes it easier for the person downloading the code to follow along. I am not as religious in the workplace, especially if there is no standard for organization in place. This is primarily to avoid restructuring.

Why avoid restructuring? There are a couple of reasons, but they primarily relate to Team Foundation Server and Team System. I imagine the same is true for any source control system, but reorgs in TFS are easier done when you rebuild a completely new project with the new organization. The problem there is you lose history. So, if you are using a source control system, you are back to leaving everything flat or having a firm, largely immutable, standard that is followed.

3. Be consistent in naming

This one is dear to heart, as I currently have to do a refactoring of class names, as some of the names we have right now are confusing or way to similar to other names.

What do I mean by “be consistent in naming”? If you are going to change the name of a class, you need to change the name of the file. It makes things much easier when searching for a file when there is a problem in Visual Studio.

As an slightly off topic subject, you should also not have multiple classes in a single file. I say “slightly” as multiple classes in a single file presents the same problem as having the file named one thing and the class named another.

I am a bit lax on project names, which was the core of the problem I was having today. BUT … and this is a big but … you need to make sure any disparity makes sense and a standard is written. For example, the project MyCompany.MyProject.SubArea.Influence.CoreTests.MSTest.Integration.dll can be shortened without much problem, as long as you follow the same scheme in all of the projects.

Summary

The main takeaways for this post are as follows:

  1. You have to take the time to refactor when you change things. This is not just refactoring of code to remove smells, but refactoring the project to ensure the naming is correct (to standard) and consistent (which the standard should cover).
  2. You need to consider both the structure on the physical drive and the structure in your solutions when you develop the standard to help with takeaway #1.

Peace and Grace,

Greg

Twitter: @gbworld