How do you measure a Program? In Lines of Code, in variables, in functions, or schedules calling.(1) The real question of how do you measure the size of a model needs to be taken back one step; what do you want to get out of your measurements?
The measurement of the model will help us to understand 5 things:
- How hard it will be to test
- What it takes to develop, how much effort to maintain
- How long it will take to execute
- Is it complete against requirements
- How easy will it be for others to interpret the model
Likewise there are a set of standard measurements that we make
- Lines of Code/Blocks in a Model: the literal counting of the number of functional blocks or lines of code. This can be improved by measuring the LOC per function or the blocks per atomic subsystem.
- Cyclomatic complexity: A measure of the total number of independent paths through a model, e.g. how many choices can you make?(2)
- Floating point operations (FLOPS): This is a count of the total number of floating point operations which can be used to estimate execution time.(3)
- Interface complexity: a measurement of how complex the interface to the model is, e.g. how many inputs, outputs and how co-related they are.(4)
- Requirements coverage: a measure of how the model maps onto requirements.(5)
| Test | Develop Maintain | Execute | Requirements | Understand | |
| LOC | x | x | x | ||
| Cyclomatic | x | x | x | ||
| FLOPS | x | ||||
| Interface | x | x | x | ||
| Requirements | x | x | x |
In the table above, the X’s correspond to the strong links between the two; in truth, all of the tests provide some insight into the objectives.
Focus on development
The use of metrics often comes late in the development cycle closer to release; however they are best employed from the start of the project. There are N reasons
- Time to completion: over time, the use of metrics can allow you to to predict when the project will be completed
- Error reduction: There is a strong correlation between metrics and bugs in code
- Modularization: Metrics should be used to determine when models should be split into multiple models for development.
Modularization can drive you off your rocker
Who you are,(6) the role you play in the workflow will influence the desired size of the models; there are a Quadrophenia(7) number of roles. In general the desire for smaller models ranges from the tester (“smallest”) to the release engineer (“largest”).
- Tester
- Developer
- Integrator
- Release
So looping round, how do you square the circle and use the core metrics to create a model development workflow that satisfies the conflicting desires? The short answer is, you cannot. As a result, you need to prioritize the members of the development team that have the greatest impact on overall product development time and quality…the test engineers and developers. This may result in models that are smaller then the integration engineers desire. However, since the highest proportion of software bugs are at the unit level, priority should be given to the unit level development and test. To make it easier for the integration and release engineers to develop, consider using a “shell model” approach to speed up the initial development workflow.
Coda
Before adopting any metrics you should be clear what you intend to measure(8) and what you want to do with that information. One company I consulted with had a “number of cups of coffee per lines of code metric”; but in the end it only showed that they had an espresso machine. On that note, remember to pick your metrics well.
Footnotes
- Perhaps one day I will release the album “The greatest hits from MBD”, vocals by my lovely wife Deborah.
- If I hadn’t used the “Choose your own adventure” image in a recent blog post I would have used that instead of the ruler for this section. Darn-it, I need to plan ahead!
- This is a tricky metric, as a single function can have multiple FLOP counts depending on the conditional path the model executes for any given cycle. Because of this, FLOP is generally reported as min, average and max.
- By related inputs, if you have 4 inputs such as Right Front tire pressure, Left Front tire pressure, etc., those are related. However the price of coffee, inches of rain in Spain, and number of ants at a picnic, those are not related.
- Requirement coverage answers three questions; first, are you meeting all the requirements for your model? Second, are you testing all your requirements? Third, and often missed, are you putting too many requirements into too small of a space? In other words, when I see a model that has 15 requirements associated with it, I start to question the validity of the requirements and the reality of the implementation.
- Yes the lyrics are reversed here, Who, are you, Who Who Who Who?
- For those of you who have seen the movie you know that in reality the four roles Jimmy plays are really one; this is much like software development where many people play multiple roles.
- Even “simple” metrics like lines of code can be difficult to fully specify. Is a line of code all of the code between the assignment (far left) and the semicolon on the “far right”? What do you do with code that spans multiple lines? Comments (inlined?)