Ah tests! Those silent protectors of developments integrity, always watching over us on the great continuous integration (CI) system in the clouds. Praise be to them and the eternal vigilance they provide; except… What happens to your test case if your test infrastructure is incorrect?
Quis custodiet ipsos custodes(1)
There are 4 ways in which testing infrastructure can fail, from best to worst
- Crashing: This is the best way your test infrastructure can fail. If this happens the test ends and you know it didn’t work.
- False failure: In this case, the developer will be sent a message saying “fix X”. The developer will look into it and say “your infrastructure is broken.”(2)
- Hanging: In this case the test never completes; eventually this will be flagged and you will get to the root of the problem
- False pass: This is the bane of testing. The test passes so it is never checked out.
False passes
Prevention of false passes should be a primary objective in the creation of testing infrastructure; the question is “how do you do that?”
Design reviews are a critical part of preventing false passes. Remember, your testing infrastructure is one of the most heavily reused components you will ever create.
While not preventing false positives, adherence to standards and guidelines in the creation of test infrastructure will reduce common known problems and make it easier to review the object
There are 3 primary types of “self test”
- Golden data: the most common type of self test is to pass known data that either passes or fails the test. This shows if it is behaving as expected but can miss edge cases(3)
- Coverage testing: Use another tool to generate coverage tests. If this is done, then for each test vector provided by the tool provide the correct “pass or fail” result.
- Stress and concurrency testing: For software running in the cloud, verification that the fact that it is running in the cloud does not cause errors(4)
- Time: Please, don’t let this be the way you catch things… Eventually because other things fail, false positives are found through root cause analysis.
Final thoughts
In the same way that nobody(5) notices water works until they fail, it is common to ignore testing infrastructure. Having a dedicated team in support is critical to having a smooth development process.
Footnotes
- In this I think we all need to take a note from Sir Samuel Vimes and watch ourselves.
- There is an issue here; frequently developers will blame the infrastructure before checking out what they did. Over time the infrastructure developers “tune out” the development engineers.
- Sometimes the “edge cases” that golden data tests miss are mainstream but since they were not reported in the test specification document, they are overlooked by the infrastructure developers.
- The type of errors seen here are normally multiple data read / writes to the same variable or licensing issues with tools in use.
- And if you look at it with only one eye, failures will slip passed.