Tejas Software Consulting Newsletter

Jason Gorman again inspires me to write, in response to his post Freelancers Are People Too. He says that most people who write code for a living are freelancers. That’s an interesting statistic, and if it’s true, I would be curious to see a source for it.

What really got me thinking was his story about a survey that one of his clients made of its employees. Freelancers were not included in the survey, and Jason claims that this was a mistake. I don’t know what the survey was about, but Jason makes a reasonable argument that freelancers have opinions that could be valuable to their clients. I’ve seen cases myself where organizations sought feedback from employees but not contractors.

But between the lines, Jason seems to be saying that he doesn’t like being treated like a second-class citizen. I don’t really share this concern. I never forget that I am a service provider for the organizations I’m working for. In fact, in the US the IRS demands that I not act too much like an employee, lest they judge that I actually am an employee and make the client pay additional taxes. There are many things going on between employee and employer that I’m not a part of, and I’m thankful to not be involved in some of them.

A contractor relationship is subtly but qualitatively different from an employee relationship. I want my clients to be successful, but I’m also concerned with making my own business successful. There’s a clear and contractually-defined line between them and me. I still get to feel the thrill of victory when a project is successful, and layered on top of that is the thrill of success in my business and the improved market perception of my services. I also know my engagement with them could vanish in the blink of a budget shortfall.

Another factor that Jason may have in mind is contractor retention. Employers take steps to encourage their employees to keep working for them, but don’t put as much effort into keeping their contractors happy. Given the prevalence of contract labor in the software industry, an investment in “contractor retention” may be worthwhile. In Jason’s case, the client he was frustrated with may have been risking an early departure from him.

I have added three public course dates to my training offerings, all via Rex Black Consulting Services. I will be teaching all of these -

Managing the Testing Process (brochure):

June 25-27, 2008 – Dallas, TX

Performance Testing Immersion Workshop (brochure):

September 22-24, 2008 – Denver, CO

October 21-23, 2008 – Austin, TX

If you sign up, please tell them I sent you!

My StickyMinds column posted today - Peeling the Performance Onion, co-authored this time with Rex Black. When I taught the Performance Testing Immersion Workshop recently, the students were surprised to learn about hockey-stick shaped performance graphs. They also learned about how performance tuning is similar to peeling an onion, and in the article I extended the vegetable theme by applying Rudy’s Rutabaga Rule. (By the way, I’m amazed that my new column is already on the first page in a Google search for “Rudy’s Rutabaga Rule.” And now even this blog entry is also on the first page.)

So anyway, we decided to share these concepts on StickyMinds. How well do these ideas fit with your experience? Please post your comments here or on StickyMinds.

Jason Gorman posted about “The Future Of TDD - Real-Time Feedback As You Type?” on his blog. He says that unit tests should give us feedback as fast as the feedback he got when dropping his security badge -

“On my way back from lunch today, I was walking down the stairs when I heard a loud “thwack” and turned round to see that my security pass had come loose from its clip and fallen to the floor.”

Wouldn’t it be great if bugs made a thwacking sound the moment they landed in our code? He goes on to imagine an IDE someday running unit tests as we’re typing code so we get real-time feedback. He suggests that if it takes five minutes to run unit tests, that’s far too long, so we’ll have to wait until computers get much faster before his idea is feasible.

But I would suggest that if we have unit tests that take five minutes to run, we’re may not be doing unit testing properly. I recently wrote a unit test for a Perl module using Perl’s xUnit - Test::Unit. There are 18 tests, achieving close to 100% coverage, and they run in about 200 milliseconds of cpu time and well under a second of elapsed time. I like Michael Feathers’ criteria for unit tests, which say that they don’t talk to a database, don’t communicate across a network, don’t touch the filesystem, don’t require configuration files, and if a test case takes longer than 1/10 of a second, it’s glacially slow (see Working Effectively with Legacy Code). I did violate one of these, because my tests do create and delete files several times; I haven’t found a good way to mock file access in Perl. But I’m happy with the subsecond run time.

It’s conceivable that a large system could have five minutes worth of well-written unit tests. Jason hinted that he’s thinking of running all of the system’s unit tests when he says that we could try to take a shortcut and only run the unit tests that are dependent on the code that changed. But that doesn’t sound like unit testing to me, it sounds more like integration testing. I do have suites of subsystem tests and system tests that take more than a minute to run, but I don’t call them unit tests. So we’re probably using different definitions of what a unit test is. I know that many people use the term “unit test” very loosely, which tends to mask the fact that people often aren’t really getting the benefits of good unit testing.

So I think that Jason’s desire for very quick feedback is closer to our grasp than he thinks. We need to test our code in extreme isolation from the rest of the system, so that as many bugs as possible will make a loud “thwack” in the unit test environment, before we pull out the slow integration tests.

Update: seeing the inelegance of “close to 100%”, I added a few tests and now have 100% branch and condition coverage for my class. But I still have much to learn about migrating my black box testing skills to unit testing.

SQE’s Joey McAllister interviewed me for the March 2008 Gray Matters podcast (click the link for the mp3 file), where we talked about the testingfaqs.org Boneyard and the strength of the worldwide test tools market.

What is an “international consultant?” Someone willing to travel to help clients in any country? If that’s the case, I’ve been an international consultant since I got my passport shortly after opening my consulting practice, just in case. But that doesn’t seem sufficient for the title “international consultant.” How about having clients in at least eight countries? But I hadn’t actually left home to do what I do for them, so that didn’t seem like good enough either.

Perhaps tonight I’m a real international consultant. I’m sitting in a hotel not far from Toronto, Canada, where I’m teaching a course. So though I’m barely across the US-Canada border, I’ve finally used my passport for something other than a pleasure trip. Sorry to dispel the image if you thought I was already a globe trotting road warrior. :-)

It doesn’t matter to me what you call me, as long as people keep calling me with interesting software testing challenges I can help with.

One of my favorite non-fiction writers has broken into fiction. The Aremac Project is Jerry Weinberg’s first novel, published in 2007. Jerry is a friend, so I can’t offer an unbiased review, but here’s my review nonetheless.

The book doesn’t look like a novel—its 6×9 inch paperback format looks like many of the technical books on my shelf. At a list price of $17.95, it’s also not priced like a novel. The book is published by Little West Press, an imprint of Dorset House Publishing, which has published many of Jerry’s nonfiction books. Dorset House also published Tom DeMarco’s novel The Deadline a decade ago, though they curiously don’t include it with Aremac in their list of novels [they have since fixed this].

Though flawed in some ways, on the whole I found The Aremac Project an engaging story with several interesting dimensions. The plot is timely, revolving around the U.S. government’s war on terror, and engaging for me because the attacks occur right in the homeland. The book does a great job of stimulating an appreciation of many cultures, especially Arabian. If you know Jerry, you won’t be surprised to find a German shepherd in the story, and she’s given a unique role.

True to Jerry’s goal of creating “nerd lit,” the book makes several references to software culture that computer professionals will recognize. One character invokes Brooks’s Law. There is a brief mention of the importance of testing. And one character extols the virtues of egoless programming, a term coined years ago by Jerry himself, though Jerry doesn’t stroke his own ego by citing its origin. The character Harvey Wyatt embodies a bumbling researcher, a pointy-haired boss, and an astonishingly incompetent programmer all in one person, and he sounds all-too-similar to real people I’ve worked with. I felt that some of these references could have been worked into the story more seamlessly instead of resorting to expository diversions.

Some parts of the story seemed implausible, such as the government’s willingness to assist a private research lab that it had no formal relationship with, and the subdued public reaction to an invention as revolutionary as the Aremac. The security built into the system was an important part of the plot, but I couldn’t figure out the motivation for some of the security measures since it was largely a standalone system. The courts seemed unrealistically enthusiastic to make use of the brand new invention, however, I appreciated several twists in the court procedures that seemed entirely realistic, even in how they didn’t seem logical to the highly logical hero in the story.

I often can’t keep track of all of the characters in a novel, and there are plenty of characters in this book to keep me confused. Some of the confusion is intentional, and I suspected this throughout the book without being able to predict what the twist in the end would be. I often didn’t get enough cues to to track the passage of time through the story. The loose ends were all tied up in a big expository narrative at the end, which seemed like too much of a literary shortcut.

Despite never being able to turn off my inner critic, I was able to enjoy the story, which offers both suspense and science fiction. The best science fiction predicts (sometimes even motivates) innovations in the real world, and in his blog Jerry recently pointed to research in brain scanning that in some ways parallels the innovations in the book. Watch for more fiction coming from Jerry soon. I’m hoping for an Aremac sequel.

Want to be a ticket taker?
Want to be a pizza maker?
Lobsterman. Jockey. TV fixer.
Baller dancer. Soda mixer.
Do you want to be an astronaut?
Or keeper of the zoo?
You’ve got to do something.
What DO you want to do?

This is the time of year that I reflect on what I want to do. The quote above is from Maybe you should fly a jet! Maybe You Should Be A Vet!, written by one of my favorite authors masquerading as Theo. LeSieg. Later he suggests that we could “run a big computer lab.” I think I’ll stick with something along those lines. :-)

This January marks seven years I’ve been in business as Tejas Software Consulting, and fifteen years of software testing. Over the last year I’ve seen a qualitative change the strength of my sales pipeline. Even in the frequent troughs in the market, sales leads have still been trickling in. Maybe it takes seven years to build a viable independent consulting practice? I still have a long way to go to be where I want to be, but it’s encouraging that I’m making progress.

I had several direct consulting and training clients, but also a growing stream of business channeled through other sources. For years I hadn’t had much luck getting work through other consulting companies and training companies, largely because there wouldn’t be room in the rates for the middlemen to take a share for their efforts. But recently I’ve had several gigs through different service providers, including more than one incredibly with three middlemen between me and the client. Some serious networking went in to those deals!

I did a fair amount of travel, coming in spurts as always. There was something about Seattle and Boston that kept me going back, for more than one project in each. And there were first-time visits to Los Angeles and Chicago (not counting excursions I made to Chicago when too young to remember). This year it looks like Toronto is the theme, with a few different events likely to draw me in that direction.

There was a great deal of diversity in the work I did last year, which I love. I was involved in several assessments, including some small organizations as well as the largest IT organization I’ve ever worked with. In a few projects, I wasn’t the leader, which was a different experience for me. I had several opportunities to teach my Introduction to Software Testing course, and I also helped create the Performance Testing Immersion Workshop, where I switched roles and had a taste of being a software product developer.

My advertising line of business on testingfaqs.org is booming, and is now more than 10% of my revenue. Despite the worries about a recession all over the news, the test tools market looks strong, judging by the advertisers who are signing up with practically no sales pitch. I get a global view of the tools market, dealing with advertisers across four continents. About 60% of my advertisers are in the US, all along the east coast, several in California, and a smattering in between, including the rare case of one in nearby Dallas. Long-time contacts in New Mexico and Florida also signed on, neither of them in the tools market that most of my advertisers play in. The percentage of US advertisers is shrinking, though. I’ve also worked with two vendors in the UK, two in Russia, plus Switzerland, France, India, and Australia.

A few other notes on the past year -

• I became a full-time Mac user, and I’m hooked. I love my MacBook Pro.
• At least once a week I get a call or email from someone in India hoping I can find them a job as a tester. Some say they’re responding to my ad, though I’ve never placed a job ad in India. They say curious things like “Please revert back to me” and “Do the needful.” Sorry folks, I’m not a recruiter.
• Somehow, I’m still coming up with new topics for my StickyMinds column.
• I’m still active on LinkedIn. I’ve been focusing on helping moderate the TopLinkedIn mailing list and growing the membership of the Ex-Convex and Metro-SQA LinkedIn groups.
• I’m seeing occasional interest from people around the world who would like to start a chapter of the Association for Software Testing, but I haven’t had the energy yet to motivate any of them to make it happen.

That’s what I’ve been up to. What about you?

I’m developing a suite of subsystem-level tests for a client. I was delighted to receive this message from a developer on the project:

I seem to be more motivated by seeing tests pass than by closing things on my bug list :-)

We talked about the how the tests seem to function as positive reinforcement, while bug reports are more like negative reinforcement. This is probably an important part of becoming test-infected. But we’re still not unit testing, and the reason in this and many other situations is the setup effort required to get a unit test harness selected, installed and ready to use.

I’m still struggling with getting test-infected for my own code that I write. I had a recent success with a library I wrote to help with testing using bisection. I started out without unit tests, but when I started to struggle with the code, I added tests using Perl’s simple and universally available Test::More module, and that made the development process much easier. Because the code was object-oriented, the unit tests were easy to write, even if I didn’t use an xUnit-style test environment. When I shared the code with my colleagues in the Toolsmith Guild, I also sent the unit tests, considering them a core part of the code as well as an example of how to use it.

This morning, I was struggling with a CGI script that’s part of testingfaqs.org. It has been spewing warnings in the error logs for eons. When I tried to add a new feature, the script stopped working, and I couldn’t get the web server to give me a useful error message. “Aha!”, I thought, this is the perfect situation for unit testing legacy code. I found Perl’s CGI::Test module, which was exactly what I needed. But it wasn’t installed on the server. So I attempted an install, but the installation program got killed because it was exceeding the server’s memory quota. Oh, yeah, my ISP doesn’t like me doing development directly on the server (shhhhh!). So I checked on my Mac, and CGI::Test wasn’t there either. Attempting to install there yielded a compilation error for one of the module’s prerequisites. And I haven’t made the effort to build a BSD Unix staging environment to more closely match the web server.

So here I am, stuck without a unit test environment, in the same situation that blocks so many other people from unit testing, and frustrated to the point of not having the energy to keep trying. Maybe that new feature isn’t so important after all.

My article Bisection and Beyond is featured on StickyMinds this week. Many testers use the bisection approach when the zeroing in on the minimal input that causes a failure, though some don’t know there’s a name for it. In this article, I describe the basic approach, and also suggest some fruitful opportunities to deviate from the basic mechanical approach.

At the bottom of the article there is also a link to a discussion about tools used to help with bisection.