Give me code or give me death...

Recursively Delete a Directory Structure

Wed, 20 Aug 2008 17:15:23 GMT

So you want to delete a directory programmatically, but you're not quite sure just how to do it? You're vaguely aware that the process requires this nasty thing called "Recursion", but the very act of uttering the word causes a negative physical reaction? Don't sweat it. We've all been there.

Yes, recursion can sometimes be a difficult topic to wrap your mind around. But the process of deleting a directory (with all of its subdirectories) is actually a great place to start the learning process. It's a great place because it's a concept that we're all already familiar with. I mean, we must delete umpteen directories every week, right? So we're not going to have to try and grapple with the problem domain and the algorithm at the same time.

There are many definitions of the general term "Recursion" (often having to do with a mathematical operation being defined in terms of itself). In Computer Science, however, we get a little more specific than that. Recursion, to us programmers, describes a situation where a function calls itself. There are some problems (like traversing all of the nodes in a tree, or deleting a directory structure) that are extremely easy to solve within the context of a function that can call itself. Those same problems, however, are often painfully difficult to solve (or horribly inefficient) if another approach is taken.

The following code is an example (a horrible example) of a recursive function:

That function does exactly what it implies: It merrily goes nowhere....forever. This is obviously a problem, right? A single call to MerrilyGoNowhere() will bring the walls of reality shattering down around us, leaving our program as little more than a steaming pile of hot rubbish. Well, maybe not that drastic, but at the very least we've just entered the shadowy-realm of the non-terminating loop. It's a place we don't want to be.

The problem is that it's a place you might frequent until you get over the initial learning curve of recursion. There must be a step, somewhere in that function, that steps us back out of the recursion and up the call stack. Let's refactor a little bit.

The key to recursion is to only recurse conditionally. Some set of circumstances must be true in order for you to take the recursive step. Otherwise, you step out of the function, which steps back up the call stack and stops your function from creating a non-terminating loop. The "Else" block above isn't technically required, but it demonstrates that you can do some other processing if your recursion condition isn't met.

Take a quick walk through the code above, step by step:
We launch our program, and a call to MerrilyGoNowhere() takes place.

On this first pass, let's say that WeHaveEnoughEnergy() returns "true", so we call MerrilyGoNowhere() again.
On the 2nd pass, WeHaveEnoughEnergy() returns "true" again, so we call MerrilyGoNowhere() Again.
On the 3rd pass, however, let's say that WeHaveEnoughEnergy() returns "false". In this case, we skip the recursive step and go straight to "GetSomeRest".
The function then returns up the call stack to the previous call (the 2nd call) to MerrilyGoNowhere(). However, no further processing is required in that function, so we return up the call stack to the previous call (the 1st call) to MerrilyGoNowhere().
Once again, we find that no further processing is required, so we return up the call stack once more. On this return, we've completely left MerrilyGoNowhere() behind, and our program can continue or shut down gracefully, having avoided a non-terminating loop.

Those are the basics of recursion, and those basics rarely change even for the most complex problems. Let's apply what we've learned here to the process of deleting an entire directory tree.

First, some groundwork. We know that in order to delete a directory structure, we're obviously going to need to delete the files in those directories, right? In fact, programmatically, we need to delete the files prior to attempting to delete the directory. Let's write a simple function that handles deleting a file:

As you can see it's a fairly simple program. It merely sets the attributes of a given file (Element) to FileAttributes.Normal and then attempts to delete that file. I've written the code in such a way that if an exception occurs, the function returns false. Otherwise, we return true. You don't have to make a separate function to handle file deletions. I just did so to clean up the code you'll see below.

Ok, so we have a function that will delete a file that we pass to it. So how do we delete the directory structure? Well, recursively, of course! Let's look at the following code:

Here we can see that recursion is our friend once again. The function takes one parameter "directoryPath", which is the fully-qualified path to the directory we want to delete.

Let's walk through the code:

We call "DeleteDirectory()" and pass it a path to the directory we want to delete.
We check to see if that directory actually exists. If it does not, we display an error message.
Get a list of all of the elements in the directory.
Begin looping through those elements. If we find that the current element we are looping through is itself a directory, then we call DeleteDirectory once more, passing it that element (which we now know to be a path to a subdirectory).
This process continues until we find an element that is actually a file. When we find a file, we simply make a call to DeleteFile(), passing it the file we just discovered.
When no additional files or directories are found, the function returns up the call stack.
The next line of code deletes the now-empty directory.

There is some additional refactoring that I'd love to do to that function (I don't like some of the names I've chosen for methods/variables, etc.), but that's the algorithm in a nutshell!

Don't Forget to Flush!

Wed, 13 Aug 2008 19:35:10 GMT

I saw something new today that I haven't seen since my C++ days: a stream that needed to be flushed.

I have to admit, this one boggled me for about 15 minutes before I realized what was going on. Then I caught a mistake I haven't made in years: I forgot to close an open StreamWriter.

The function is a simple one. It iterates through decorated methods in my Program.cs file, finds the command name through reflection (thanks to a colleague of mine), and streams the command name to a text file for later review (after some collection manipulation and sorting). Here was the original method:

As you can see, it looks pretty innocent. Except for one small problem: In a collection of 60 strings, only 44 1/2 of the strings were being streamed! Yes, you saw that correctly. Only 44.5 of the 60 total strings were being written to the text file. Then I had a C++ flashback that threw me into a post traumatic stress episode...

Back in C++ we had to routinely "flush" our streams to make sure all the data made it safely to its destination. I don't ever recall having to explicitly flush ANYTHING in C# or, dare I say it, VB.Net (<gasp>)! Then I looked back at my code and slapped myself square on the face.

I forgot to close the stupid stream :( 15 minutes down the toilet because I forgot to flush (pun intended). In C#, making a call to StreamWriter.Close() causes an implicit flush to take place. No, I don't know or understand the details yet. I just know it happens.

With a single line of code: sw.Close(); I was able to successfully stream all of my commands to the text file. Here is the finished product:

Not much of a difference in implementation, huh? But a HUGE difference in effect.

VSMDI...Shmee-SMDI

Mon, 04 Aug 2008 21:05:38 GMT

In our day-to-day development efforts we often find ourselves confronted with an issue screaming to be dealt with by excessive force. Anyone who has spent any time with Microsoft's VSMDI file can empathize. If you have more than one developer on the project, you are guaranteed to run into the dreaded "multiple VSMDI file" issue. It often happens with only one developer, for that matter.

From an application developers perspective, this isn't much of a problem to have to deal with in many circumstances. From an automations perspective, however, it can quickly become a nightmare. In automations, we need to be able to predict the environment we're executing in, or at least be able to establish a certain state that we can rely on. The VSMDI file allows neither of those solutions to work. So what are we to do about it?

Ignore the VSMDI file.

Say what?

You heard me. Ignore the VSMDI file. You don't need it. The only thing you need is the assembly that contains the unit test(s) you are interested in. You're probably thinking "so you expect me to maintain a list of every unit test assembly in the project? Are you out of your mind?" No, and Yes, respectively, but the latter is neither here nor there.

All you need to know is the topmost parent directory of your application (the root of all directories that could possibly contain a unit test directory...without going too far up the directory hierarchy. It does no good to go beyond the scope of your application, so why bother?) Once you have that directory, a simple recursive search will do all you need it to do.

The following C# function will recursively search and find any assemblies that match the search criteria that you specify. In the case of my organization, any assembly that contains unit tests actually contains "UnitTest" or "UnitTests" in the file name, so that makes my search much easier. I also exclude any "obj" directories, or any directories that happen to be named for the service account that runs our automation (any directory with the name of the service account was most likely generated by previous test runs, and will contain assemblies you don't want to re-run). [Sorry about the spacing...still trying to figure out how to best post formatted code on LiveJournal.]

private void LocateAndExecuteUnitTests(string RootDirectory)
        {
            String[] FileSystemEntries;
            FileSystemEntries = Directory.GetFileSystemEntries(RootDirectory);

            foreach (string Entry in FileSystemEntries)
            {
                if (Directory.Exists(Entry))
                {
                    // Directory found. Need to recurse (if not the \obj directory).
                    if ( (!(Entry.Contains(Path.DirectorySeparatorChar + "obj")) || (Entry.Contains(Path.DirectorySeparatorChar + "AcctName"))))
                    {
                        LocateAndExecuteUnitTests(Entry);
                    }
                }
                else
                {
                   // Restrict execution to only those assemblies that contain "UnitTests" or "UnitTest" in the file name.
                    if ( (Entry.ToUpper().Contains(".UNITTEST.DLL")) || (Entry.ToUpper().Contains(".UNITTESTS.DLL")) )
                    {
                        ExecuteAssemblyWithMSTest(Entry);
                    }
                }
            }
        }

Once I've located a valid assembly file, I pass it to ExecuteAssemblyWithMSTest like this:

private void ExecuteAssemblyWithMSTest(string PathToAssembly)
        {

            StringBuilder sb = new StringBuilder(string.Empty);
            sb.Append(" /testcontainer:" + PathToAssembly);
            sb.Append(" /publish:" + StringConstants.TFS_SERVER_NAME);
            sb.Append(" /publishbuild:" + LatestNamedCIBuild);
            sb.Append(" /flavor:" + "\"" + StringConstants.OA_BUILD_FLAVOR_CI + "\"");
            sb.Append(" /platform:" + "\"" + StringConstants.OA_BUILD_PLATFORM_CI + "\"");
            sb.Append(" /teamproject:" + "\"" + StringConstants.TFS_CI_TEAM_PROJECT + "\"");

            try
            {
                Process proc = new Process();
                proc.EnableRaisingEvents = false;

                proc.StartInfo.WorkingDirectory = StringConstants.OA_PATH_ROOT_DIRECTORY;
                proc.StartInfo.FileName = StringConstants.MSTEST_PROCESS_NAME;
                proc.StartInfo.Arguments = sb.ToString();
                proc.StartInfo.CreateNoWindow = false;
                proc.StartInfo.UseShellExecute = false;

                proc.Start();
                proc.WaitForExit();
            }
            catch (Exception ex)
            {
                logger.LogException(ex, true);
                throw ex;
            }
        }

The above function builds a process object and then executes that process. MSTest is invoked (via the process object), and ultimately publishes the test results back to our TFS server.

Voila! No more VSMDI files!

Quality Assurance does NOT begin with the QA Team.

Wed, 30 Jul 2008 20:37:41 GMT

In my experience in software engineering, I've found that many organizations claim to be "quality-centric", or may have even achieved a certain CMM level. In reality, however, these organizations rarely QA their products beyond simple manual test cases designed and implemented by a QA team (when the organization in question even has the luxury of a QA team). These organizations believe that unit tests (generated by Visual Studio), combined with a manual test process, will shake any existing bugs out of a system. They are correct in thinking they will find bugs in the product, but not for the reasons they think they are correct.

Most organizations wait until the end of a development cycle to begin thinking about QA. They schedule a certain period of time at the end of the cycle to test and debug. Yes, they are going to find bugs. Moreover, the fact that they will find and fix bugs during that "phase" of development will have the overall effect of convincing that organization of the correctness of their approach. "See! We TOLD you we'd find bugs!" These organizations tend to suffer from one of the "Self-fulfilling prophecies" mentioned by Steve McConnell in his book Code Complete ("We'd better start coding right away because we're going to have a lot of debugging to do."). Incidentally, if you don't have a copy of that book yet, run to the nearest book store and buy it. Anyone who calls themselves software engineers today should have a worn-out copy of that book on their desk. But I digress....

The primary function of quality assurance is to avoid introducing bugs to begin with! In the event that bugs are introduced into the product, then the secondary function becomes detecting and fixing that bug as close as possible to the time it was introduced. It is possible, not to mention highly desirable, to produce defect-free code. Don't let anyone tell you otherwise! No human endeavor is perfect, but that fact is often abused by developers who are, quite simply, too lazy (or burned out) to implement quality processes in their day-to-day routine. So, how do we avoid introducing bugs to begin with?

The first step on this path is recognizing that quality assurance does not begin with the quality assurance team! It begins with the individual developer at the keyboard who is trying to decide whether or not to expose a certain method on a class, or whether to refactor a certain block of code into a method of its own, or whether or not the class itself exposes a consistent level of abstraction. (We could easily get derailed here by discussing upstream activities. And while I agree that much of the individual developer's success is determined prior to his even being hired, I'm going to take an "all else being equal" approach to this subject and concentrate exclusively on the individual engineer).

We've set the stage, so now let's talk about some specific things a developer can do in his/her day-to-day routine to increase the quality of their product. Here are a few of my favorites, in no specific order...

Agree on a set of coding standards early in the process
At the end of the day, it doesn't matter whether you use Hungarian Notation, or decide you're going to preface all of your Int32 variables with "George_", as long as your entire team/organization agrees on some standard and you document that standard. Discussion about coding standards can quickly devolve into mini holy wars, so I'll try to avoid that. Suffice it to say that the purpose of a coding standard (within the context of Quality Assurance) has little to do with the quality of the standard itself. The purpose of the coding standard is to ensure that all developers, both present and future, understand how to interpret code they have before them. Code that follows a strict set of conventions goes a long way towards achieving that lofty goal of self-documentation.

Review all code during development
Many technical leads are reduced to nothing more than code "traffic cops" in certain organizations. They go through a typical day from one code review meeting to another code review meeting, peering at sample code and trying to keep the development teams on the proper coding standard path. This is a very limited view of code review, and it turns code review sessions into interrogations where developers feel they must defend every underscore and semicolon. That is not the sort of "defensive coding" we want our junior developers to employ! Code review serves three primary purposes:

Catch poor design implementation before other coding efforts become dependent on the poor implementation.
Enforce proper coding standards and naming conventions, etc.
Educate all developers (not just the junior developers).

Many organizations I've been associated with focus so much on item #2 above during review sessions, that they miss fundamental implementation problems, and they miss countless opportunities to educate their developers. Well-planned and properly-executed code review sessions will have a dramatic impact on the level of quality in your product. In fact, well-executed code review sessions are so beneficial that the impact is not solely experienced by the current project! Well-executed code review sessions increase the overall body of knowledge of your fellow developers (team leads included), which has a beneficial impact on any future projects your team tackles.

TDD: Test Driven Development
There was a time when writing tests was exclusively the responsibility of the QA team. But it has been shown, time after time, that developers who are savvy about test development routinely produce a higher quality product. Before I talk about what Test Driven Development is, let me address what Test Driven Development is not.

It is not Tester Driven Development. It is not the purpose of the test team to decide what should or should not be developed. Those decisions should flow naturally from the body of project requirements.

Test driven development begins by developing a test. That sounds a little counter-intuitive, until we dig a little further under the hood. By developing your unit test first, you are making a statement about the class you have yet to implement. You are defining the context within which "success" can be claimed. In other words, you are explicitly stating the conditions under which a particular class can be considered "successfully implemented". The first time you compile this code, the test should fail...because there should be no class implementation yet. The theory is that you develop the class until such time as the test passes. When the test passes, you are done. That class is now feature complete, and you can move on to other classes.

This process is valuable for a number of reasons:

It forces the developer to understand the class before he/she implements the class!
It catches most non-integration defects at the time they are introduced, which is the optimum time to catch any defect.
It avoids waste by letting the developer know when they are "finished".

Number 3 above is very important. I've lost count of the number of times I've seen a good implementation go bad because the developer didn't know when to quit. Being a perfectionist comes along with the territory of being a software developer. Developers, by nature, tend to have a higher attention-to-detail than most other people. But this can become a liability if it isn't kept in check. We often desire to implement "enhancements" (or whatever we want to call them) in an effort to make our class perfect. We tend to lose sight of the fact that perfection is defined by our requirements, not our personal goals.

"Stop the Line!"
I've recently been reading a book named Implementing Lean Software Development From Concept to Cash by Mary Poppendieck and Tom Poppendieck, at the behest of my employer. This is a very interesting book for any software engineer who is interested in implementing more efficient processes that have withstood the test of time. The book documents the amazingly efficient manufacturing processes of Toyota, and how those processes can apply directly to software development. One of the principles (amongst many) the book recommends is taking what the authors call a "Stop the line" approach to development.

At Toyota, the company has given each individual employee who is involved in the manufacturing process the ability to stop the production line at the first sign of a defect. The line comes to a complete halt, and no further production takes place until the cause of the defect is discovered, and a solution is put in place. This one seemingly-obvious concept has dramatically reduced the number of defects that pass through the quality assurance process and ultimately end up in the hands of consumers.

At Thermo Fisher Scientific, we have translated this process into our own iteration-based development cycles. When a defect is detected, development stops, and we find the cause of the defect before moving on to implementing more features. We have gone so far as to declare a build "broken" if any single one of our hundrends of unit tests fail. What this means for us is that we cannot consider a feature implemented if it contains a defect of any kind. The cost of moving forward will simply be too great if we hold off on fixing our defects until some future point in time.

Manage Complexity
Steve McConnell, in his book Code Complete, reiterates several times (himself referencing an earlier author, whose name and work eludes me at the moment) that the prime directive of software engineering can be summed up in two words: manage complexity. The way in which we make a quality product is to implement it in the simplest fashion possible. One of my pet peeves as a software engineer rears its ugly head whenever I see code that appears to be complex for complexity's sake. There was a time when it was laudable to be as cryptic as you possibly could be. He who fit as many operations as possible into a single line of C/C++ code was the winner.

Then we, as an industry, started adding up the costs associated with software development and came to realize the folly involved in that approach. Obfuscation is a noble effort when the goal in mind is security or protection of a valuable intellectual property. But we should never obfuscate something that is intended for consumption by other developers! We obfuscate our code in subtle ways that fall far short of direct encryption, but nevertheless have a similar impact. We obfuscate our code every time we use a "magic number", or create an inheritence hierarchy that resembles a pictogram of the Tree of Life. We obfuscate our code whenever we create a series of sequential operations, but fail to document the sequence. We obfuscate our code whenever we see that a method has a cyclomatic complexity measured in 4 digits, but fail to perform even the most basic refactoring. We obfuscate our code when we insist on abbreviating all of our variables, even though the abbreviation only makes sense to us. And the list goes on....

The list above represents just a small fraction of the things we can focus on to increase the level of quality in our product. What I find interesting is that the guidelines for increasing product quality also seem to be common-sense guidelines to becoming better software engineers.