For those of you reading in the distant future, e.g. more than a month from now, let me set the stage. It is July 2018 and (as an American) World Cup Soccer is in full… More
I have written about reuse in a number of post before; today I wanted to talk about a conceptual framework of reusability. In this video I talk about four types of reusable “objects”
- Concepts: These have a high reusability with a corisponding high initial development cost. Examples include:
- Model Architectures, testing infrustructure, modeling guidelines
- Fundemental objects: These also have a high reusability, but have a realitivly low initial cost. Examples include:
- Transfer function implimentations, bang-bang control algorithms…
- System implimentations: These have a medium level of reuse with and a moderate level of initial implimentation cost. The “smaller” the system is the lower the cost. Examples include:
- An internal combustion engine model, a wing’s control surface
- Targeted implimentation: These have the lowest level of reusability and implimentation cost. Examples include
- Hardware device drivers, specific fault montering algorithms.
For complex systems, the goal of “complete” testing can be impossible to reach. This often boils down to issues of time and complexity. This is the most “common” type of failure. A second, more insidious type of failure is improper testing.
What is improper testing?
Improper testing arises from not fully understanding what is being tested or how the testing algorithms work.
Let’s take a simple example, response time to a step input. For this example, we will use the following test requirement
TR_1028: Within 0.5 seconds of the unit step input the control system will settle to within 5% of the input value.
Failure type 1: Single point pass:
There are two errors with this test:
- The hard comparison for the driver input value. A correct method for evaluating this would be
abs(driver-target) < eps(target)
- The pass condition is a single point in time. What this means is that the signal could continue past the target not settling.
Failure type 2: “Noise” intolerant
To fix this error the settling time needs to be defined. To fix this a new state of “Settling” is added. However, we have now introduced a new error
- Hard failure condition: the response two the signal can include overshoot before the signal dampens down. The current test fails if the signal falls out of the “less then 5%” range at any time.
Solution to this issues:
The solution to this issue is to define a max settling time and to trigger a failure condition if the signal has not settled within the time.
Failure type 3: Sampling
A common failure mode, when dealing with large data relates to sampling. Let’s take a look at an error flag example.
TR_2034: During the duration of operation the check engine flag will not be active.
It is common when logging data in a system to downsample the data; e.g. log every 10th or 100th data point. If the flag is intermitent it is possible that the flag would not be logged.
There are two solutions to this problem; first, the code could be refactored to include an “ever high” flag. The second option would be to reconfigure the sampling rate for this specific test.
Failure type 4: Matching problems…
One common test type is baseline comparisons. There are three types of problems that arise when comparing baseline data
- Failure to align/sampling: baseline data is often taken from physical systems or prior simulations. In this case, alignment of the data between the simulation and the test data is critical. Likewise, if the sample rate of the baseline and the simulation are different then the alignment becomes more difficult.
- Poorly defined tolerances: comparisons can be made against, absolute, relative and percentage error. Further, the data type (integer, enum, double) have an effect on the type of comparison that should be performed
- Scaling/Units: The final, common, baseline data issue is a failure to correctly account for differences in units and scaling between the to sets of data.
The types of failures described here are a subset of improper testing issues. They represent a failure to understand the full scope of the behavior of the system or a failure to understand how the collected data can be inspected.
In today’s video I answer the questions
- How do I learn Model-Based Design
- What is the “key” technology for Model-Based Design
- How did I get into Model-Based Design
Recently a client posed a question I have heard a number of times, “How many States can I have in my model before there are problems”? On the surface, this seems like an O.K. question, however, when we dig in a little we see the inherent assumptions with the question.
Size matters, after a fashion
As a basic metric, the number of states in a model is meaningless, it is akin to the question “how many lines of code before there are problems”? If someone said they had a program with one function and 100,000 lines of code you would assume that it was problematic in its complexity. On the other hand, if they said the program had 100 functions you would think that the model was well architected. Going to the other extream if the function had 1,000 functions you may think that they have created architectural problems of increased complexity.
No one builds a house with a Swiss army knife
Models are tools, they perform functions in response to inputs. It is possible to build a single function that performs a 1,000 different functions but that is rarely the correct way to go.
Rather each model should be viewed as a specialized tool performs a function or set of related functions. Again this relates to the “100 or 1,000” functions for a 100,000 lines of code. I generally consider something a “related function” if
- Uses the same inputs: E.g. the function does not need to import additional data
- Is used at the same time: E.g. the information is used in the same larger problem you are trying to solve.
For example, calculating the wheel speed and wheel torque in the same ABS braking function makes sense as they use the same input data (generally a PWM encoder) and are used at the same time (to determine the brake pulse width). However, calculating mileage in that function, which can be derived from the wheel speed, does not make sense as it is not part of the same problem you are trying to solve.
Keeping it in memory…
In this instance, I am talking about the developer’s memory. Above a given size and complexity it becomes difficult for a developer to remember all the parts of a function operate.
As a general rule of thumb, I try to stick to a “depth of 3” limit. No subsystems or nested states more than three levels deep. If there is a need for greater depth I look to see if there is a way to decompose the model or chart into referenced models and charts. One note, when measuring “depth” the count stops when a referenced model or chart is encountered as these are assumed to be atomic systems developed independently from the parent.
Benefits of decomposition
The following benefits are realized through the decomposition of models
- Simplified testing: large functions have a large number of inputs, outputs, and possible responses. Smaller models have reduced testing criteria.
- Simplified requirements linking: Generally, well decomposed aligns with the requirements by not clumping disparent functionality together.
- Improved reusability: Smaller functions are more likely to be generic or easily customizable.
- Improved readability: A smaller model can more quickly be reviewed and analyzed then a larger model.
What is the correct question?
There are two questions I would ask:
- How do I make the model functionally correct?
- How do I make the model readable?
For guidelines on that topic, you can read my Stateflow Best Practices document.
The following is an idealized Model-Based Design workflow, from initial requirements to product release. The workflow assumes a multi-person team with resources to support multiple roles.
It all starts with requirements…
Ideally, the process starts with foundational tools and processes in place. These consist of
- Modeling guidelines: Covers model architecture, data usage, and model clarity
- Testing guidelines: How the system will be validated
- Requirements tracking: A system for tracking the compliance and changes to requirements
- Bug reporting: A system for tracking issues as they are discovered; this is tightly coupled to the requirements tracking.
- Support infrastructure: The supporting infrastructure includes tools such as version control and CI systems.
- Project timeline: The project timeline provides the objects for the completion of the project and the resource allocation (e.g people)
From the initial high-level requirements derived requirements and derived tests are defined. These documents, along with modeling guidelines and testing guidelines are used to create the initial system and component level models and their associated test cases. These models and tests are entered into the requirements tracking system and assigned to the appropriate teams.
During the development phase. both the models and tests points are elaborated by adding in details from the requirement and testing documents. As these features are added to the system the models are validated against the test cases.
As issues are uncovered, either way in which the design is lacking or bugs are discovered information is added to the bug and requirements tracking systems. This workflow goes on in parallel for each developer/model.
This development goes on in parallel for both the system under development and any supporting models such as plant and environmental models.
System integration and release
As the maturity of individual components matures the system level integration and validation rigor is increased. At this stage of development testing of the derived objects, e.g. generated code, increases in frequency and rigor.
After the release of the project, the components from the development should be examined for reusability. Beyond components processes and guidelines should be reviewed for updates to the best practices.
Plant models, whether based on first principals physics or regression models based on real-world data are a cornerstone of the Model-Based Design controls development process. During the initial development of the control algorithms a “static” physical model is sufficient; however when the development moves into the diagnostic and release phase physical models that demonstrate real-world variation are required.
Variations, not noise…
In a previous blog, I wrote about the importance of noise in testing. Variations are different from noise in that they are a constant offset. For instance, my height will always be 6’3″ while my wife Deborah’s will be 5’10”. If we design an airbag system assuming everyone was 5’10” then there could be issues when the first 6’3″ person is in the car with an accident.
Working with variations
If we continue the “body variations” example and think of all the variables associated with the body, height, weight, leg length, arm length… we will observe two things
- There is a correlation between some variables: In general leg length increases as height increases, as does weight.
- There are outliers: While there are general correlations between properties there are still outliers which cannot be ignored.
So given these two considerations how do we proceed?
Data at the boundaries, data in the center
Test data should be defined that includes both data at the boundaries and in the ‘center’ of the test space. Data at the boundaries exercises the edge cases while the data in the center is used to validate the mainline behavior. When considering which boundary conditions to include consider the following issues.
- For discreet variations: In instances where the variations are discreet, e.g. on/off, flow/no-flow all discreet instances should be included
- For continuous variations: In the example of height, values at the endpoints should be selected along with a set of points within the range. (The total number should be a function of what a nominal unit is in the range. For instance, if we took a height range from 4’10” to 6’6″ and assumed the nominal unit of 1″ then perhaps a spacing of 6″ would be reasonable)
Variations and variations… working with multiple variations
In any real-world system, there are multiple parameters that will vary. Selecting which combination of variations (outliers and central points) needs to be determined in a rigorous fashion. In an upcoming post, I will cover how six sigma style approaches can be used to determine which points should be selected.
Interface Control Documents (ICD) are a method for specifying the functional interface of a component or system. Used correctly they prevent integration errors and promote formal development practices.
What is in the document?
At a minimum, the ICD consists of the following information
- Input specification
- Data type
- Output specification
- Data type
- Global data used
- Calling method
- Periodic or event-driven
- Reusable / non-reusable
The I/O and global data are generally well understood. Specification of the calling method is required to understand how time-dependent functions such as integrators or transfer functions will behave.
Additional data may include information such as signal range, update rate, units… All of this information should be derived from the requirement specification. (Note: the ICD is sometimes viewed as a derived requirement document)
How is the ICD used?
The ICD provides a baseline for the interface to the software component. Once the ICD has been defined the initial model can be created. This is sometimes called a “shell model.” The shell model has all of the inputs and outputs as defined by the ICD document. The shell model can then be integrated into the system level model (or another integration model) for a system lockdown. This integration model provides the first level of testing of the interface. If the interface of the shell model changes the integration model will break.
As I have written about in previous posts I recommend the use of reusable test utilities. When working in the text-based MATLAB environment how to create reusable utilities is easily understood; they are simply MATLAB functions. However, within the Simulink Test graphical environment, it may not be as clear.
Libraries and Functions
Fortunately, there is a solution; if there wasn’t there would be no post today. Within the Simulink Test environment, calls can be made to functions. The functions can be either return a value (or values) or directly set an assert or verify flag.
The functions are imported from a Simulink Library and can be constructed from MATLAB or Simulink Function blocks.
In the case of MATLAB functions that are placed in a Stateflow block with the functions export option selected.
So there you have it, a simple solution to reusable test utilities within the Simulink Test environment.
Are you in or near Huntsville AL? Would you like to meet me and have a chance to learn more about The MathWorks and MathWorks Consulting? Well then come out to the MATLAB Aerospace and Defense Smart Systems Tech Briefing.
One of the rationales for adopting Model-Based Design is an expected Return On Investment (ROI). This has three very natural questions
- What is the expected ROI?
- What is the timeframe for realizing the ROI?
- What is necessary to realize the ROI?
Unpacking the ROI questions
The first thing to recognize is that the ROI will be dependent on the “level” of adoption of Model-Based Design. The more processes of Model-Based Design that are used the greater the ROI, however, there is a corresponding delay in the realization of the ROI (see reference 1).
Further, the ROI is dependent on a having a defined implementation plan. A full MBD process includes multiple tools and tasks, without a well-defined implementation plan the dependencies between these tasks will become muddled.
Assuming a well-defined implementation plan, Most companies will start to see a return on investment after 9 months to 1 year. The majority of the ROI is generally realized after 3 years.
Hidden or “Negative” ROI
One aspect of Model-Based Design makes measuring ROI difficult, the fact that model-based approaches allow for the development of systems that are impossible (or at least extremely difficult) to develop using traditional approaches. In these cases where MBD is used to create systems of high complexity, the measured ROI may be lower than actual ROI due to the inherent complexity of the system.
Finally, what is the expected ROI? From industry examples, ROI’s as high as 80% are known to be possible (see reference 2) with ROI’s of 30~40% are considered common. Again, these results are dependent on having a good implementation plan. Hopefully, this blog, or MathWorks, will help you develop that plan.
- What is the benefit of a model-based design of embedded software systems in the car industry? By Manfred Broy Technical University Munich, Germany
- Measuring Return on Investment of Model-Based Design By Joy Lin, MathWorks
- Model-Based Design in Practice: A Survey of Outcomes for Engineers and Business Leaders. By Dr. Jerry Krasner Chief Analyst at Embedded Market Forecasters