Test plan development. Regression analysis. Systems documentation creation. Test case execution. Regression testing.

If you're anything like me, then those words may as well have been boring, tedious, mind-numbing, tiresome, dreary, and the-worst-thing-in-the-world. Sure, they're all important and necessary, but you found out that, due to budgetary constraints, you couldn't personally do any of those things and could only focus on coding, you probably wouldn't complain. Julien G. certainly didn't mind, especially since the "drudge work" would still get done by the overseas team while everyone state-side was fast asleep.

Naturally, many of the other developers were upset that their work was being sent overseas. Emotions ranged from annoyed to absolute outrage, and some were angered to the point of resignation. Eventually, things cooled down and developers bought management's pitch that "the world-class, high-quality engineers" would be good for "synergizing and strategizing the bottom line."

Grouped Together

The offshore developers were split apart into different teams. There was one for infrastructure, one for database and data integrity, and another for working with the application's User Interface. This latter group was the one that Julien worked with.

Their job, in part, was to act as any dull, average customer and regression test the application. If at any point the application crashed or behaved strangely, they were to document the conditions leading up to the crash and send them back to with details of how the problem was recreated.

At first, things were rough. Julien spent many a late night with the offshore team, walking them through their hellishly complex application. But Julien felt it was worthwhile, as it forced him and his group to be more organized and provide more exact specs and concrete test cases. Soon enough, the offshore team was able to jump into the application and provide the right information back to developers, and the developers were sending back recompiled libraries to test with as quickly as possible.

As a result of the growing synergy, productivity was way up, morale was up, and — to the delight of management — the bottom line cost was way down. From all angles, everything looked like puppies and rainbows. That is, until it was time for Offshore Integration Phase 2.

Phasing Fail

With all the initial pains of getting the offshore team trained on using their software, it was time to put their "world-class" engineering skills to use. In addition to running loads of manual regression tests, the offshore team would be responsible for developing automated unit and regression tests. This, of course, meant that they'd need to be given access to the code and set up a testing environment at their location.

Being as large and complex as it was, their application wasn't the type with "run the installer and follow the wizard" installation instructions. In fact, there were 85 steps necessary to complete the installation, ranging from installing the database to editing configuration files. It wasn't all too difficult, as installing the application on a local workstation was often the first task given to newly-hired developers.

Julien sent these instructions to the offshore team and, the following morning, arrived to find a disappointing email.

Hi Julien,
After the program start-up reaches 45%, it quits unexpectedly.
Please confirm that latest code is the most recent.

Regards,
Ravi M.
Implementation and QA Lead
Hyderabad Group Inc. 

After some re-verification, Julien confirmed that he had, in fact, sent the latest code. Just to make sure the install instructions were valid, he spent a good two hours installing the application on a brand-new server. The only thing he could guess was that some third-party component wasn't properly installed, so he sent an email back to Ravi to verify all components.

Hi Julien,
All components installed as per instructions. We followed instructions
three times, each time with a freshly formatted server. 

It still quits when the program start-up reaches 45%. Please advise.

Regards,
Ravi M.
Implementation and QA Lead
Hyderabad Group Inc. 

The email routine continued, each time with the offshore team insisting that every instruction was followed perfectly. After a few weeks, the offshore managers had their weekly conference call with state-side managers and in doing so, accused the developers — specifically Julien — of blocking the offshore team's progress. With things getting heated up, Julien was charged with the task of watching the offshore group perform the install and configuration step-by-step, via a shared desktop across a sluggish network connection... at 3:00AM his time.

Just Following Orders

It was only about 4:30am when Julien was on his third cup of coffee, asking the offshore team to confirm step number 32 of 85 on the install and setup list. Julien was in the midst of wondering how feasible it might be to set up a recording of him saying "un-huh - confirmed' to go off at set intervals when he noticed something amiss in their copy/pasting of environment variables from the setup document.

"Hey guys," Julien jumped in, "did you do anything different this time with the setup? Like right now?"

"No," the offshore engineer responded, "we executed the steps exactly as prescribed."

"Okay, Julien said, "can you scroll back up a little bit in the configuration file?"

The offshore engineer complied and, located near the top, the following code stuck out:

' Environment variable for test provider
' 
set PATH_TO_TST_ENV=Path to the test environment (absolute, such as C:\TestEnv) 

The line was exactly as it had appeared in the setup document — copied verbatim — beneath a heading that read: "Please recreate the application configure file precisely as listed below"

At this point, Julien could have died from shock if he weren't so tired. Baffled, he asked why they didn't substitute the path with the directory they had created in the previous step. "This is what you instructed us to do," the offshore engineer replied, "we followed exactly what was in the documents to avoid making any mistakes."

It was an unfortunate "ah-hah moment" for Julien; not only did the offshore engineer's response explain the issue at hand, but it explained a whole host of otherwise unexplainable and unreproducible bugs. After that day, Julien learned to be a lot more specific and careful with what he wrote. And while his days are no longer filled with running test cases, he's found he spends an awful lot of time testing and "debugging" the test cases, which, somehow, takes several times longer than executing them.

"I've heard about verified by Visa, but this is something new," writes Velmu.

"Well then," Simon Timms wrote, "it might take some time to fill in all those arguments."

"I guess a 25% price increase is a good reason to be wowed," Steve Frein writes.

"I was recording some drums with my keyboard, and all of a sudden this came up," Jonathan Flusser wrote, "apparently it has to do with the Trash, and it's very hard to do!"

"I cancelled my free trial," notes Frederick Ding, "but I'd love to be paid $10/month for using 0 MB."

"I saw this at a pay station in a parking in Barcelona, Spain," Riccardo Zanussi writes, "I don't know if I really want to put my credit card in this machine."

"I found this in a book about iPhone programming," writes Mike Kerley, "the rest of the book seems to have received an equal amount of attention to detail."

"At first I thought this was just another way to ask me to enter 1," Nick S. W. noted, "but then it popped up when I entered 1. Is there another integer I'm forgetting about?"

Ryan Schlesinger writes, "I went looking for information about the iPhone release for Telus and found this instead."

"This left me in a bit of a quandary," wrote Dominic, "to install, or not to install?"

Not too long ago, I was at a client site, working to understand and improve their development process. From a birds-eye view, their development organization was a lot like many other Corporate IT set-ups: they had a sizable portfolio of proprietary applications that were built for and used by different business groups. Some of these applications were “mission critical” and had highly formalized promotion and deployment processes, while others were ancillary and were hardly ever used. <shameless_plug>This, along with the medley of technologies and platforms, was why they sought our help in managing and automating their development processes with BuildMaster.</shameless_plug>

But as I dug deeper, I noticed that a significant portion of their applications weren’t applications at all. They were – for lack of a better word – “modules” that glommed together to form an ÜberApplication. Completely unrelated business functions – paid time-off tracking and customer mailing list management – lived side-by-side, sharing authorization principals, navigation controls, and even a “business workflow engine.”

Digging even further, I learned that most of these module-applications were derived from a “one-size-fits-all base application” of sorts. For example, the back-end paid time-off system was nothing more than a calendar with a custom UI for manipulating “events” (i.e. PTO requests). The front-end consisted of simply displaying these events to anxious employees who were gladly counting down the remaining days until a vacation from maintaining this amalgamation of a system. Other module-applications were “document managers”, “news posters”, or some other universal sounding name.

The whole thing may as well have been replaced by a pre-alpha version of Sharepoint or Google Docs. It was as if they had tried to build a skyscraper with Erector Sets. And not real Erector Sets, but some poorly-made knock-off.

The developers absolutely hated the ÜberApplication. It took longer to “customize” a template than to build an application from scratch, it was harder to test and deploy, and they never quite fit the business requirements The architect (who was the second successor since the original architect)  despised it as well, and eventually convinced management to ditch it for constructing new applications from scratch.

Sadly, this was not the first time I’ve seen this set-up/architecture. At one of my first jobs, we had something comparable, except much less formalized and much more disorganized. And there are all the examples I’ve shared through The Daily WTF, but this recent experience got me thinking: how exactly do well-intentioned development organizations end up with horrible systems like this?

I See a Pattern Here

We human beings are quite remarkable at recognizing patterns. Take clouds, for example. A cloud dog looks nothing like a real dog, yet no matter how hard we try, once we see the dog in the cloud, that’s all we can see.

While this ability has clear evolutionary advantages, it’s often a disservice in today’s modern world. Pattern recognition yields many false positives, leading towards Gamblers’ Fallacy, prejudice, and can even extend to really poorly-written software.

Now, recognizing patterns at the micro level (i.e., code) is almost always a Good Thing. Code often does repeat itself, and consolidating repetitive code into subroutines tends to help throughout development and especially when it comes to maintenance. The real problem – and the one behind the aforementioned systems – is recognizing patterns at the macro/application level.

A Pattern of Failure

As completely different as Paid Time-Off tracking and Customer Mailing List management may seem on the outside, they do share quite a bit in common.

• Navigation of some kind that links between major and minor functions in the application
• Authentication and authorization for users of the application
• Pages/screens that view and/or edit a single record
• Lists that display multiple records based on some search criteria
• Et cetera, and so on

In fact, if we kept going, the list of similarities between applications would grow much larger than the list of differences. But that doesn’t mean that they’re basically the same thing. Just as our canine friends share an astonishing amount of DNA with us, it’s our differences that make us so unique. The same holds true in software, and forgetting that fact will inevitably lead towards the dreaded Inner-Platform effect.

This is precisely how so many organizations end up with their own ÜberApplication. Consider, for example, that same portfolio of applications I described earlier. At one point, the applications evolved normally: i.e., they were built from scratch following development guidelines to suit the specific needs of the business client. And they looked something like this.

It’s hard to describe that portfolio as anything but chaotic, let alone accept that it’s actually how things should be. Let me repeat that last bit. A disparate application portfolio is a good thing. Proprietary software has a high strategic value to the organization, and building it in a manner that doesn’t meet the requirements largely defeats the purpose.

Still, it’s so easy — and so tempting – to forget that last part. Each new application built felt like reinventing the wheel, especially when they all had some variation of the same components: authentication, authorization, navigation, databases, etc. The rules for each of these applications were vastly different — a simple password required for one application, Active Directory group-based authorization for another — yet the designers had that overwhelming urge to abstract and “simplify” the process of creating new applications. This started at the requirements level by simply mandating that all applications share a universal set of requirements for certain things.

The uniformity was found in the UI: the layout must feature a header, a sidebar, and a footer, but the business customer can pick out the background color. It also was in the form layout: labels should be placed above form fields, and lists should always be sortable. And the database: there was to be a users table, groups table, roles table, etc.

Eventually, the applications started to blend together. Worse, the requirements conversation shifted from “how can we build software to meet these needs” to “how might we adapt the needs to meet our pre-determined requirements.”

Once the universal requirements had been defined, the next logical step was to abstract them into some sort of framework. After all, Duplication Is Evil, and it doesn’t make sense to re-implement the same User-Group-Principal-Role-Task-Operation security in each and every single application. Now, in addition to being uniformly developed, the applications were all dependent upon yet another in-house built codebase: the Global Application Framework. Granted, the GAF was little more than a wrapper on top of a framework (.NET), but somehow, it was perceived to be better.

The leap from here to the ÜberApplication wasn’t far. After all, application metadata was duplicated across several points: source control, server configuration, global navigation, and so on. Removing these points of duplication, along with consolidating all of the applications in one location, brought us to where we started.

Avoiding the Path to Inner Platform

The Road to WTF is almost always paved with good intentions, and there are few intentions more noble than making developers’ lives easier. Of course, given that developers’ lives are pretty easy as is, and there are a whole bunch of companies who build developer products, it’s pretty hard to improve, especially when your development department is just a cog in a large corporate machine. In fact, all too often, the opposite effect happens, and the “innovation” becomes one of the biggest obstacles.

Most of us developers embrace the Don’t Repeat Yourself/Duplication Is Evil principle, and we apply whenever we can to the software we create. But one of the deadliest pitfalls we can make is waving the wand of abstraction too quickly and too broadly, and we often forget that the consequence of unnecessary duplication is far less than the consequence of unnecessary consolidation.

Problematic Problem (from Ben)
Way back when, I was responsible for doing on-site support for a fairly complex ERP solution that our company sold. My support radius was 100 miles, which meant I was on the road a lot and traveled to places I wasn't all that familiar with. My trusty navigation aide was a compass and a Rand McNally map book. Fancy, online mapping services weren't around yet, let alone super-fancy GPS units.

One day, I was assigned to visit a customer on the far end of my region (99.9999999 miles), first thing in the morning. It meant that, not only would I need to battle rush-hour traffic through the city, but then drive another 60 miles once that cleared. I was not a fan of early mornings, and getting that client on that wintry day meant a 5:30A departure with a 2.5 hour commute.

That morning, traffic was even worse than I anticipated. And to make matters worse, I had a terrible time finding the place. Fortunately, a kind fellow at the gas station pointed me in the right direction, and I was able to ring the client from the nearest pay phone to let them know I was running behind.

When I arrived, everything seemed to be downhill from there. I went to the receptionist, tacked on my visitor badge, headed over to the server room, set my briefcase down, and got to work. Before I could even try logging in, someone walked up to me and said, "hey, I know this isn't really your thing, but I'm desperate, and reeeeaaaallly need some help getting this report for our PM meeting."

It certainly wasn't my thing, but given that I was 30 minutes late, a little goodwill towards helping a company executive could only help. So I followed her to her office and helped troubleshoot the problem. An hour-and-a-half later, we had the report running, no problem. She was thrilled, and I headed back to the server room.

For some reason, I couldn't log-in to the server console, but the generous IT guy helped me past that hurdle by logging in with his credentials. But then I had another problem: I couldn't access any of the servers listed on my sheet. In fact, I couldn't even find a server that looked anything like ours.

I called the IT guy over again and asked him where our ERP server was. He shot a confused look to me, and said that he's pretty sure they don't have an ERP server. I assured him that they did, so he went back and looked into things on his end. Thirty minutes later, he assured me that they absolutely, positively, definitely don't have an ERP server.

We were both utterly confused. And then something dawned on me, and I silently prayed it wasn't true. I pulled out my sheet, showed it to the IT guy, and pointed towards the customer address heading. "That's you guys, right?"

As it turned out, not so much. Our actual customer was down the street, in another un-marked office building.

Problem supply (from Brendan)
Working as a coder for a small company that operates worldwide, I was on the team that deployed a project to China. Now I realize that my English is far from perfect, but dealing with Chinese customers in English has been quite the experience. One day, four months after going live with the new system, I received this mail from our Chinese client:

From: Louis Chang
To: Brendan ******
Subject: Problem supply
_____________________________________________________

Hi Brendan,

Sorry disturbing you. There is a problem with supply programme on the
button. Please advice?

Regards,

Lou Chang
 

Ah, the lingo of the busisness... I can imagine that you'd have the faintest idea what he was talking about... but don't worry, neither did I. So I replied to him, hoping to get a better description of his issue:

From: Louis Chang
To: Brendan ******
Subject: RE: Problem supply
_____________________________________________________

Hi Lou,

Could you please state your problem more clearly?

Thanks,
Brendan

I didn't have to wait long for his clarification, as his problem seemed to be really urgent.

From: Louis Chang
To: Brendan ******
Subject: RE: Problem supply
_____________________________________________________

There is a problem
with supply
programme on
the button.
Please advice?

Yup. Much better. Thanks.

A Text-Destroying Problem (from Esko Tanakka)
Back in 1999, I was just beginning my career and worked at a small store that built and configured computers for the public. Occasionally, I'd have to answer customer calls and help people with general computing problems.

One day, a man called in and immediately started complaining about how we sell utter crap, that we should take responsibility for our problems, and that he was owed money back because of the problems we caused.

I begged him to calm down and explain specifically what was wrong. He told me that our computer is destroying his text, and that something had to be done. At first, I thought his files were disappearing, but after more investigation, I discovered what his actual problem was: typing text in Microsoft Word overwrote previous text.

I told him that he simply had the INSERT key on, but he insisted that he never pressed that key, and that pressing the key did nothing. Running out of phone-support options, I told him he’d need to bring in his computer. But first, I needed his warranty information.

Well, it turned out that he bought the computer seven years earlier, then had another company install Windows 95 and the Corel Office Suite. After hearing that, I told him that I obviously couldn't take the machine in. That just made him more angry, and he accused me of working for "Satan and his minions", and threw all sorts of other ridiculous insults at me. But then all of a sudden, he calmed down. Apparently, he actually tried pressing the INSERT key (as I asked him to do before), and his computer stopped destroying his text.

"I guess the contact lens service that sent me this package couldn't read the label," wrote Mike Totman, "Maybe they should double check their prescription."

"Coming up next: someone will 'type words here.'" James Oakleywrote ,"Sounds exciting!"

"While playing Counter Strike: Source, I received this error," wrote Nick C " I sure wish I knew a programmer that could fix this."

"Okay," wrotes Serg, "first part? Check! Second part? Check. Third.... wait, WTF?"

"I guess Microsoft isn't quite used to success," writes Eric Packwood

This message popped up in Infrarecorder," notes Dan H, "I'm not sure why they were asking me, though."

"I think we have the harshest password policy ever," noted Devin Cameron.

"That's one way to say it," writes Marcus.

"Excellent!" writes Daniel, "Some Ass is available in HD!"

"I saw this a few months back," Scott Selikoff notes, "apparently, saving 0 cents is pretty spooky!"

"Unable to decide how much of a discount to give to random spam recipients," noted Michael Hoskins , "Pfizer takes the 'shotgun' approach to discounting."

"This code was left by the Senior Software Consultant," Michael Wheeler writes, "I'm not sure if it's insurance against 'Return' not returning... or a comment that explained why the line of code was left in."

Public Shared Function GetItemFromValue(ByVal ddlControl As DropDownList) As Integer
    Dim i As Integer
    If ddlControl.Items.Count > 0 Then
        For i = 0 To ddlControl.Items.Count - 1
            If ddlControl.Items(i).Selected() Then
                Return i
                Exit For     'leaving this here cause we're in a rush
            End If
        Next
    End If
End Function

"We're aparantly pretty serious about 'properly' handling exceptions," writes Rami, "very, very serious."

public bool Receive( string FilePath )
{
    try
    {
        return true;
    }
    catch( Exception e )
    {
        ExceptionManager.Publish( e, Priorities.High );
        return false;
    }
}

"A nice example of ahead planning," Robert writes, "just in case the Earth's mass dramatically changes overnight, we are prepared for a fast, system-wide adjustment."

Public Function Newton()
    Newton = 9.81
End Function

"Some developers use JavaScript for validation, others use server-side code," Rio writes, "we, on the other hand, seem to use comments. It doesn't really validate anything, but hey, it's good."

If fromDate < toDate Then
   'it's good.
Else
   'it's not good.
End If

"I was going through some obscure code from the developers before me, and something caught my eye," Philippe wrote. "It was a class whose task was to transform data, and this is how it was instantiated."

Transformer optimusPrime = new Transformer();

"Gee thanks," Tobias N. Sasse writes, "that's helpful!"

public long getLength() {
  return 1000000;  // no clue
}

"I tried to put some italic tags around a bit of text today," writes Derek, "but somehow, it emboldened my text. I did some digging, and I found that our front-end developer has this in our CSS stylesheet."

i { color: #000; font-weight: bold; font-style: normal; }

"I guess the comment header is right," Brian M writes, "no po box for you!"

//*******************
//**
//** Function   - POBox
//** Purpose    - no po box
//**
//*******************
 
function POBox(eobj, eid) 
{

	return (true);
}

"So that's how you DeUnicode stuff" David Nguyen wrote, "and here I thought it involved a bit more than removing an italics tag."

function DeUnicode($_input){
  //added to strip out italics tag <i> from name
  $_input = str_replace('<i>', '', $_input);
  $_input = str_replace('</i>', '', $_input);
  return $_input;
}

“Our codebase is a bit... backwards, to say the least,” writes Aaron Silver, “things that should go up don’t go up or down... instead, they’re painted orange .”

“The postProcessAddress address method is a good example of all of this.”

string[] months =
{
 "Jan", "Feb", "Mar", 
 "Apr", "May", "Jun", 
 "Jul", "Aug", "Sep", 
 "Oct", "Nov", "Dec"
};

if (processingClient.StreetAddress != null
  && string.IsNullOrEmpty(processingClient.StreetAddress.Line1))
{
  for (String month : months)
  {
  if (processingClient.StreetAddress.Line1 != null
    && processingClient.StreetAddress.Line1.Contains(month))
  {
    writeError(processingClient,
      errorList,
      "Street address line 1 is in date format ("
        + processingClient.StreetAddress.Line1 + ").");
  }
  if (processingClient.StreetAddress.Line2 != null
    && processingClient.StreetAddress.Line2.Contains(month))
  {
    writeError(processingClient,
      errorList,
      "Street address line 2 is in date format ("
        + processingClient.StreetAddress.Line2 + ").");
  }
  if (processingClient.StreetAddress.Line3 != null
    && processingClient.StreetAddress.Line3.Contains(month))
  {
    writeError(processingClient,
      errorList,
      "Street address line 3 is in date format ("
        + processingClient.StreetAddress.Line3 + ").");
  }
  }
}

Aaron adds, Aaron adds, “I’m not exactly sure what thought process lead to this, but the powers that be aren’t too keen about changing it, leaving our end users to adding random hyphens and spaces for those unfortunate enough to live on ‘October Road’.”

“I was hired as a ‘best practices consultant’ to help bring a 300-developer company’s development practices into the 21st century,” wrote Ian, “and after six months, I had failed.”

“Our first objective was to introduce automated unit testing. They had all sorts of horribly interconnected code, and the tests would help reduce the fix-here/break-there problems. However, after many, many tutorial sessions with developers, and quite a few long meetings spent trying to convince them of the benefits, no tests emerged. The developers stubbornly held that testers should test code, not them.”

Ian continued, “Adding some teeth to our policies, we set-up a continuous integration server that emailed everyone reports of unit test code coverage. This way, managers could take responsibility for getting their teams to write unit tests. That seemed to do the trick: the number of unit tests and code coverage started to steadily climb on all projects.”

“I finally felt that all my efforts were worthwhile,” he added, “the overall health of the team’s code would now increase immeasurably. Less bugs, less time manual testing, and all that good stuff. And then I started to look at the unit test code.”

public class StaticDataRequestTest {

    @Test
    public void startClientReqest() {

        try {
            new StaticDataRequest().onData(null);
            assertEquals(
                " processing client static data request  ",
                true, 
                true);
        } catch (Exception ex) {
            assertEquals(
                " processing client static data request  ",
                true, 
                true);
        }
    }
}

Ian added, “I guess we got what we asked for.”