Ok, Software Testing is not in my job description, but because there are no tester or QA in our development firm, every developer has to do his testing by himself, consequently we only have automated tests, and as SCMer I need formalize them as part of build automation, therefore I need to understand a bit more in it.
Software Testing is an complicated and time-consuming discipline, what I'm addressing here will not be entirely correct, but rather to be just my understanding at moment. Write them down to allow me to revisit and revise in the future.
All testing were done by our developers with all automated, hence they are all white box testing.
Unit testing verifies the functionality of a specific section of code at the function/class level, it is the most fundamental one, we use it from first year of computer science study. But unfortunately it is still the only test that some of our projects have, and in a few projects they don't even have a proper unit testing.
Unit testing frameworks help simplify the process of unit testing, it provides assertions, exception handling and other control flow mechanism that make your work of adding unit tests become relatively easy. We use Boost Test Library for C++ and NUnit for C#, some of our cool kids start to use google test.
One of the task I was working on is the Code Coverage, it tests how your source code been covered by your unit tests. In other word unit tests test code, and coverage test measure unit test. We use gcov for C++ projects (with lcov as an extension to generate reports), and NCover for C# projects.
Three common metric we get from code coverage are:
- Function coverage - How many times each function in the program been called during exercising unit tests.
- Statement coverage - How many times does each line been visited.
- Branch coverage - Does every edge/decision/condition been executed?
Unit testing is a Dynamic testing, there are other Dynamic testing such as Integration Tests, System Test and Acceptance Tests. We don't have most of them, and due to the nature that our software are in-house only software, our end users are acting as tester and doing all these tests for us kindly.
Contrast to dynamic testing there is static testing, or as we called it - code review, primarily it is just to walkthrough the code manually. In my previous company we were required to have a "buddy check" before doing any commit, now in my current one we are not that restricted but people will still ask someone in the same team to review. This does help the collaboration by letting other team members to understand what your are working on, thereby improve both the overall quality of code and the developers' skills.
ReviewBoard is the tool that I deployed to help pre-commit review, it is a relatively new open source tool so don't expect to be that perfect. I also configured post-commit mailer for Subversion to emails code to reviewers automatically after checkin is made.
Static testing without human analysis is called static code analysis, this is performed by an automated tool. Coverity is one of the tool that identifies security vulnerabilities and code defects, some say it is a very powerful, but it is very expensive as well - charge by per line of code it analyse. I wonder if it's worth because I only had a glance on it, for next few days I will take some time to deploy and test it.
There are other two statis analysis tools we use for our C# projects, they are both from Microsoft and they are free (uhh?). FxCop analyse .NET programs that compile to CIL object code, and StyleCop analyse C# source code to enforce a set of style and rules. Although FxCop was running there for a long time and its result appeared to be similiar to what we got from Coverity, but no one had ever care about them. Because they are from Microsoft and they are free, hence most of our developers think they are weak? what a pity.
These are just the tip of the iceberg in the entire software testing empire, but they covered most of what SCMer need to know and take care with. If your boss ask you to do anything more than tests that can be automated, you should suggset him/her to hire some more experienced tester to setup a testing team.
Remember, as the hub of the wheel we help but we don't do everything.
