Dear all,
I am a strong believer in the value generated by unit-tests. We programmers are not as great as we would like to be. We introduce subtle bugs everywhere (if not obvious ones), we make a lot of copy-paste errors, we make a lot of bad-replace errors, and we claim to understand the code much more than we actually do. Not to mention the fact that our memory is not as good as we think it is; when we don't make mistakes, we simply don't fully understand the code, even our code. Our mind is not that powerful to keep the ever-more complex code structures in our head.
You can see this even from the development of Sparrow. While trying to speed things up, I constantly ignored the testing aspect of development. And, I've often found that I've spent more time investigating subtle bugs than actually developing Sparrow features. That means, I've spent more time because of the lack of testing than I would have spent on writing unit-tests.
Another challenge that delayed my adoption of unit-tests was the complexity of the testing problem: it's hard to create good unit-tests for large abstract syntax trees, where small changes can have ripple effects. Simple unit-test are simply a waste of time. I need more evolved unit-testing methods. That's why I'm using property based testing. This is a method of testing inspired by QuickCheck library from Haskell, and it's based on checking the postconditions of the tested code is run with randomly-generated inputs. I'm using RapidCheck C++ library as my testing framework.
Just to exemplify the power of property based testing: During some internal training we made at my company, I've asked two groups of good software engineers "to write unit-tests for a stable sort algorithm" (in two consecutive years). Then, I've made an implementation of stable sort that contained 5 intentional bugs. Most of the unit tests that I received didn't catch any of these bugs, and no unit-test submission would catch more than 2 bugs. Some bugs were simply not caught by the sum of all unit-tests received. Then, in 15-30 minutes we were able to code a set of unit-tests that would catch all these bugs, and moreover it would (almost guarantee) that there are no bugs.
Of course, not all the problems are as simple to test as stable-sorting.
I know that the Sparrow code is particularly not easy to unit-test, but I know that we can easily increase the confidence of the code to a much higher degree. I've already found some bugs, and clarified some of the properties of my code.
So that's what I've been doing for the last month or so. I simply realized that I cannot change the complete type system without some basic form of validation that I'm doing the right thing.
To a bug-free code!
LucTeo