The following dramatization is for education purposes only:
- Me (as Actor 1): “did you run the test suite before you checked in your code?”
- Actor 2: “No, it takes too long to run.”
Cue dramatic music…
I understand where Actor 2 was coming from despite the day I lost to debugging their issue; to run the full test suite on their local machine (which would have exposed the issue) would have taken 2 hours during which they would not have been able to perform other tasks. At the same time the CI process, due to heavy loading, would have taken roughly the same amount of time. They took a common approach of verifying their component worked without examining its impact on the rest of the system; but for them as an individual, the productivity cost was too high. This could be seen as a case of unaccounted externality).
Identify your bottleneck: Moore’s Law is not enough(2)
Often the first response when tests are running too slowly is to “throw more computational power” at the problem. While this may be part of the solution it is not sufficient for a large organization’s testing woes. Rather, identifying which parts of the process are running slowly, or unnecessarily so should be your first step
Bottleneck #1
The bottleneck with perhaps the greatest impact is the failure to optimize testing routines. While correctness and robustness of test routines must be the first priority, speed of execution needs to be a close second. For example, several years ago I was working with a company on their test suite and they had a comparison function that was used in 70% of all of their tests. Their implementation used a for loop as part of the comparison operation, which was ~6% slower than the vector based operation in MATLAB.
When I pointed out the speed difference, they thought they would see a 6% speed improvement in their test cases by making the change; in reality they saw close to an 11% speed improvement due to the way the same function was used in multiple locations. If you are using MATLAB for your development environment I would recommend using the profiler features to identify where you have unnecessarily (3) slow code.
Bottleneck #2
Before jumping in the shower I stick my hand under the water to test the temperature.(4) I do not look at the weather forecast, check my calendar or see if the stove is still on. In other words, I run the test that is required for what I am currently doing. There are two complimentary approaches to reducing the testing load: dependency analysis and meta data tagging.
- Dependency analysis: In this instance tests are associated with a model and then the model’s dependencies are determined. If any of the files that the model depend on change then the associated tests are run.
- Meta data: In this case a test is considered to be of a given “tag/type”; tests of the given type are run as a batch. The tags can be related to a system (such as engine or battery) or a common task (like standards checking).
Bottleneck #3
The final bottleneck is unneeded or redundant tests. Sometimes this is when tests exist in the system for projects that are no longer active; sometimes it is when there are redundant tests. Regular pruning of the tests can take care of this issue.
Footnotes
- One day as a self challenge, I am going to try to post Model-Based Design using a Patter Song format. “I write the intro Models for the modern MBDer”
- Moore’s law should really be called Moore’s Observation.
- Unnecessarily is an important word. Remember, if you have a choice between robust & correct versus fast, then choose robust & correct. Or better yet, find the cases where you can take the fast path and those you cannot and run the correct routine for the situation.
- This should be considered a safety critical test; too hot and you burn yourself, too cold and well then, your shower isn’t good.