Test Driven Development more than doubles the lines of code you have to write. With all that extra code to write, where will we ever find the time?! We have deadlines!
Lines of code has always been a bad metric; why bring them up now? Error-free robots, programming at a constant rate, might have to be concerned. But people are neither error-free or program at a constant rate. Does it matter that the LOC count doubles? The time consuming parts of programming are: thinking, problem solving, and confirming solutions.
In my article The Physics of TDD I compare two programming techniques: Debug Later Programming and Test Driven Development. I model how TDD shortens the time to mistake discovery to near zero. When you are immediately notified of a mistake, you can fix it immediately. If that same defect went undetected, as it does in DLP, it would lay dormant waiting to cause trouble.
DLP has less code (about half) and more bugs, while TDD has more code (about 2x) and less bugs. When I do TDD I find that I make subtle mistakes quite regularly. Many times an hour, but find them immediately. Each of those mistakes would have cost me, or someone, future find and fix time.
Let’s try a little thought experiment. I’ll write production code for a couple hours without TDD. Let’s go easy on me and say I only mess up 5 times per hour. After two hours of code writing, I manually test the new production code (which takes time), and almost fix all the problems. For the fun of it, say that one mistake every two hours goes undetected, and is later reported as a bug.
What does it cost to find and fix that buried mistake? Well that depends on the mistake. How long does it take you to find and fix a bug? Tough question to answer.
According to a defect study at Hewlett Packard (described in this paper), defect repair times have a distribution as described in this table:
| Defect % | Time to find and fix | Time % |
|---|---|---|
| 25% | 2 hours/defect | 8% |
| 50% | 5 hours/defect | 40% |
| 20% | 10 hours/defect | 32% |
| 4% | 20 hours/defect | 12% |
| 1% | 50 hours/defect | 8% |
It’s just one study, but let’s roll with it. Being very conservative, let’s say the bugs left behind are the easy ones, and only take 2 hours to find and fix. If that 2 hours of effort could have been prevented by writing two hours of TDD style unit tests, I’m even! Actually, I am ahead of the game because I would not have spend that manual test time. If any of the defects were more difficult to find and fix (like the other 75% of the defects), I am way ahead of the game!
Doubling the lines of code does not slow me down.
TDD will slow you down while you are learning it. Also, I am not saying TDD will prevent all bugs. You will still have bugs, but fewer of them.
Something like this observation has been with us since at least as early as Royce’s 1970 paper. The one where he introduces the idea of a “waterfall” process only for the purpose of explaining why it does not work and not to do that (shame the Software Engineering textbook authors didn’t read past page 1).
He says that if you have a process that puts testing at the end, then when—not if—a test fails (a good thing in itself: no-one ever learned anything from a passing test) an unknown amount of unplanned rework has been injected into the project. Unknown effort to diagnose the failure, unknown effort to fix the defect, and (by induction) unknown effort to test all over again.
His recommendation is to start testing as early as possible and run testing in parallel with development (in parallel with design, in parallel with analysis, in parallel with …)
Yeah Keith! Sometimes I think we need to grind on what Royce actually said every chance we get. It may be the (at least, one of the) most misunderstood bits of software engineering lore. (Another one being using a manufacturing engineering & quality model for software, thereby, tarnishing the interest in engineering & quality ideas from a design perspective.)
A nice paper on early appreciation for iterative and incremental approaches to development is the Basili and Larman paper, “Iterative and Incremental Development: A Brief History,” from IEEE Computer 36:6:47-56, June 2003. Larman’s book, Agile & Iterative Development: A Manager’s Guide, has a lot which amplifies that article, as well.
I guess this means that you get a full set of regression tests & documentation of expected behaviour at no extra cost? As a Schedule Owner I would have thought that “something for nothing” is a very desirable property 🙂
Hi James–
Does TDD really double lines? Well, usually I find that I have maybe a bit more lines of test code than production code when doing TDD, so on the surface it looks like the lines are doubled.
But every (Java or C++) system I’ve encountered that wasn’t built using TDD has been overly bloated. I’ve seen a couple systems where the production code shrunk to a third of the original size, and a couple more that shrunk to about half of the original size, after programmers added lots of tests and started refactoring. And just looking at the bulk of systems out there, I’d take the challenge that I could reduce their line count easily by 25% and more often by 50%.
So it seems like almost a wash to me: system without TDD, twice the amount of code it needs. System with TDD, half the production code plus a comparable amount of test code. Well, of course, that compares someone who knows TDD with a bunch of programmers who didn’t give a hoot. I figure someone who was good with TDD could probably produce a much cleaner design, even without tests.
I’m working on a smaller codebase now of about 30,000 lines. After about 40 hours investment, I’ve eliminated perhaps 3,000, and there’s no end in sight of potential cleanup. I’m having a blast.
Jeff
I agree that Lines of code (LOC) has always been a bad metric but many organizations still use it for bidding purposes. Maybe some day “story points” will be the way to bid but they are still yet another fuzzy widget. When using LOC the real metric should be WDLOC (working deliverable lines of code) no claims of “error free”. The cost metric is the time to produce a WDLOC. Which for embedded systems, I would say a TDD approach would come out cheaper overall. Pay me now or pay me later. The “unit” test code itself should never be counted as LOC that is just a cost of producing the working deliverable code along with debugging and SW/HW integration.
Quite an intersting topic. I found out that, in some organization, the average LOC developers write a day is around 10. The time for writing 10 Lines of code is trivial. Even though it is doubled, it is trivial too. Most of the time is spent on debugging and finding solutions for fixing the bug in a safe way.
TDD doubles LOC. That’s true. But the test code does not get into target. So, no physical problem. Also the time for doubling the code is just trivial but the code quality is improved dramatically so that the totoal time can be saved. LOC is doubled but it is at least free and you may get considerable profit from it if your are skilled with TDD.
Hi, James!
Thank you for a great article. There are some problems with it, however.
Firstly, you don’t take into account the time wasted when a design turns out to be substandard or inflexible and you have to redesign all you impacted tests or delete them and start again.
you presume someone will write code for hours before writing tests when doing debug-later: this is wrong. We write tests whenever we’re happy with therewith, often on a minute-by-minute timescale.
It is an established fact that TDD takes more time than debug-later: many papers have established this. If you simply go slower of course quality increases (to a point). The benefits of TDD are simply because it’s slower, not better.
For example, despite your later comment that TDD doesn’t find all bugs, you don’t compare the bugs that TDD leaves unfounded (and later reported as bugs) with that that debug-later finds. Thus, the comparison is unfair.
@shmoo, thanks for reading my blog and sharing your opinions.
Have you had this first thing happen to you? It happend to me 12 years ago where tests, that were not refactored were a maintenance buden. I’ve tried to keep my tests refactored to reduce this problem.
As far as coming up with a design that is inflexible… If you know good design you can TD your way to it. If you do not, then it is not TDD’s fault that your design is inflexible. Many novices skip the refactor step. Hence bad tests with duplication that are hard to change when the code under test evolves.
I suspect the persons in this situation, unless it is a straw man, were not so experienced in design and made a mess. It’s my experience, and that of many experienced people doing TDD that TDD guides design. See these two recent articles: http://blog.wingman-sw.com/archives/382, and http://blog.wingman-sw.com/archives/415.
Regarding your second point, it sounds like you are doing good things by writing the tests just after the code. There is nothing wrong with that. I think some call that TAD. I think TDD is superior, but if you have good results with TAD and you keep the feedback loop tight, I see not problem. You don’t seem to be describing Debug Later Programming. From my experience, DLP is the most popular way to program on earth. That is what the comparison is about, the way most people program compared to TDD.
As far as it being an established fact that DLP is faster that TDD… hooey.
As far as TDD not finding all bugs… it finds and prevents a significant class of bugs. TDD is great at finding code that is not doing what the programmer thinks it is supposed to do. In DLP, when code is not doing what the programmer thinks, the programmer is often unaware. In TDD you immediately get a failing unit test. You’ll have to go find those problems, usually without the support of tests. As far as any other bugs, we could call it a draw, but we’re way ahead to start with. Try TDD for a few weeks and see the rate at which you make mistakes, and then tell me that instant notification of mistakes does not help you remove them quicker.
Finally, if after TDD and some bugs linger, the TDD programmer has tests to prevent unwanted side effects. If you are writing thorough tests as you code, you’ll have them too. But if you are doing TAD, you are in a smaller group than the masses doing DLP. Those doing DLP are likely to injecting new bugs, at least some of the time. According to RB Grady study, 25% of defects are from unintended side effects. The new feature works, some random thing is broken. I think my comparison is fair.