[This blog post is a “reprint” of one of my Linked in posts while I am on vacation] In meetings or in your office the humble whiteboard(0) is a rapid prototyping environment through which we… More
During the exploration phase of adoption, the adoption team should have become familiar with the fundamentals of Model-Based Design e.g. model architecture, data dictionaries, testing, formal methods, report generation and so on. In that this blog is focused on groups adopting MBD it is a fair assumption that the current state is “introductory” (1) however the following sections can be used to identify where additional support is needed.(2)
The following tasks should be understood…
- Encapsulation of models: How to define each model in a format with defined interfaces and data.
- Integration of models: How to integrate individual models into a larger integration model
- Integration of existing code artifacts: How to integrate artifacts from the models into existing code artifacts and/or integrate code artifacts into the models
The following tasks should be understood…
- Creation of data artifacts for use in the model: How to create data that the model can reference
- Management of data artifacts: How to store and reference data artifacts in a scalable way.
- Harmonizing data between model and existing code artifacts: How to reuse data between models and existing code artifacts
The following tasks should be understood…
- Creation of test harnesses: How to create a test harness that will exercise the model in a “stand alone” method.
- Creation of data: How to create the data used by the test harness both for input and output evaluation.
- Creation of test evaluation methods: Creation of methods for evaluating the outputs from tests
The following tasks should be understood…
- Basic use of version control: The users should understand which files need to be placed under version control.
The obvious question arises “how do I move from “introductory” to “ready?” There are three primary methods
- Training: There are multiple training courses (industry) out there allowing you to learn about Model-Based Design (university).
- Papers: Both academic and industry papers exist to help you learn about MBD.
- Outside help: Outside help can come from either hiring people with MBD experience or hiring outside consultants.
- All of the above…
(1) Having an introductory level of knowledge of Model-Based Design methodologies is not the same thing as an introductory level of knowledge of software development. Most groups that adopt MBD have a strong software development background.
(2) Most of the “Why is my state so [XXX]” I understand. However, I don’t know why Pennsilvina and Connecticut got “haunted” as their “Why.”
What will and what were you measuring?
During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured. However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).
First a word of warning, metrics are useful but they rarely provide the full picture(2). That being said there are metrics that can be monitored
Bugs found at stage X…
One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process. However, one side effect is the observation “we find so many bugs when following an MBD process.(3)” Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process. To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…
cost = Severity * StageConstant
Test and requirements coverage
Model-Based Design allows users to track the requirements and testing coverage through automated tools(4). With respect to test coverage; there are two types of test coverage. The first is requirements based test coverage; do tests exist to cover the requirements. The second are formal metrics coverage such as MCDC coverage.
The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.
Integration and development time
The final “primary” metrics are the development and integration time. The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases). The integration time is a bit more interesting.
In an ideal workflow for Model-Based Design, there is an integration model that is created at the start of the development cycle. Individual developers check their models against that integration model to ensure that there are no conflicts. Hence in an ideal flow, the integration time should be near zero.
However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes. Like the bug detection finding integration issues is done further upstream in an MBD process. Again the metric should use a weighted value based on the stage of where the integration issue is found.
This post covered what can be measured, not how to measure them; this will be covered in future posts. Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.
(1) For this post, I am assuming that you do not currently have metrics in place. If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development. Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process. While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.
In this video I briefly cover how to solicit information from the users of a new process. As I have written about in the past, buy-in from end users is what determines the success or failure of the process.
In previous blog posts, I have gone into some depth on testing, architecture and data management for models. With this post, I will cover how these three activities for the foundation of any Model-Based Design process.
Foundations and core competencies
At later stages in the adoption of Model-Based Design
processes task-specific groups will emerge (development, verification, and systems.) However, at the start of the Model-BAsed Design process users from all groups need to determine the “common language” that will be used to develop their project (1).
Architecture and data sure, but why testing?
The identification of architecture and data as foundational concepts is generally well understood. Combined they define how people will develop the model through interfaces and clear communication. So why testing in the trinity? It returns to the core concept of Model-Based Design that models are at the center of the development process. To ensure that the model can be used consistently through the development process they need to be “locked down”(2) with test cases.
Driven by this objective, the testing environment is designed at the start of the development process. The requirements of the test environment should be addressed within the architectural and data infrastructure. The good news is that best practices for the three “legs”(2) of the MBD stool are already in the well defined; it is a matter of honing them to your specific project and environment.
This post is not intended to be technical; rather it is to remind us as we develop new processes to start out with the “best path forward” from the start. In the section about the validation project, I will discuss the next round of tools that are commonly adopted.
(1) I may start using this graphic as my tag for MBD adoption. Crawl = investigate. Walk = initial. Run = validated. Flying = optimizing.
(2) Tests cases are elaborated as the model is developed. The “lock down” is achieved through the use of a continuous build and test server.
(3) The metaphor of a tripod or stool can be overused. But, to push it one last time, this is your stepping stool to the next round of MBD tools and processes. Build it well and it forms a strong foundation.
The first thing I need to clarify in this post is, what is and what is not a bug.
A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
A software bug is not is
- Incomplete features: During the software development process features will be under development(1). As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
- Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software. Again, as long as the lack of the feature does not introduce errors it is not considered a bug.
Incomplete features should already be tracked in the project planning timeline. Desired features should be incorporated into the requirements document for the next generation of the project.
Severity of bugs
Not all bugs are created equal; defining the severity of bugs is necessary for prioritizing the correction of the bugs. There are two common metrics for determining severity. Frequency, how often does the bug occur. Impact, when the bug occurs what happens to the program. The following lists provide examples of how frequency and impact could be defined(2).
- Infrequent: Happens in less than 1% of the executions of the program in the normal work tasks.
- Common: Happens for 1 ~ 5% of users in the normal course of work.
- Prevalent: Occurs for 10% of users in their normal course of work.
- Low: Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
- Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
- High: Bugs that crash the program or create a loss of data for the user.
Once the frequency and impact have been determined the ranking of the bug can be defined.
The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).
Bug fixing workflow
There are multiple commercially available tools for bug tracking. A basic workflow should include the following tasks and events.
- Bug detection: The bug is found either through use or captured in an existing test case
- Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
- Assignment: The bug is assigned to a software engineer for resolution
- Creation of test / validation of solution: If a test case does not already exist for the bug it should be created. The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.
Bug tracking and resolution is a problem common to all software development workflows. The process for resolving these issues is the same for Model-Based Development as in traditional C development environments. The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.
(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project. See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development. Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics. Without the entry there is no method for determining the efficiency of the overall process.
The question, dreaded or loved that all engineers face is, “Tell me, what do you do?” As a controls engineer, I fall back to the following example.
Me: Imagine you are driving down the highway and you want to pass someone. What do you do?
Imaginary person: Well I would shift over to the left lane and speed up.
Me: And if they started to go faster as you tried to pass them?
Imaginary person: Well I guess I would speed up some more.
Me: And after you pass them, what then?
Imaginary person: Well I would move back over and slow back to my initial speed(1).
This simple example serves as a starting point for explaining the fundamentals of control algorithms.
- The “objective”: This is the thing we want to control. In this example, it is the speed of our vehicle.
- The input: How we affect the thing we want to control; for an automobile it is by depressing the acceleration pedal that the vehicle is commanded to go faster.
- The feedback: The measurement of how close we are to our desired objective; e.g. are we going fast enough to pass the other car(2).
Everything is Newton’s method…
From the starting point of the automobile example, we can explain that the goal of a control algorithm is to drive the difference between the desired and the actual values to zero. We can expand the driving example to drive our point home.
- Overshoot: When you stamp on the gas and end up going 80 instead of the 75 you needed to pass…
- Fault detection: (Manual drivers only) when you try to hit 80 but are still in 2nd gear…
- Adaptive controls: You drive more carefully in downtown Boston(3) than on a rural highway…
As you could tell, this post is intended to be more light-hearted; however, I do find that thinking about controls problems in a non-technical relatable fashion helps me understand what I am talking about.
“It begins as an idea,
it ends with math(4)“
(1) We know this is an imaginary person since they said they slowed back down after passing.
(2)In this example we actually have an indirect measurement, are we passing the car not the actual speed of the vehicle.
(3) At least I hope you allow more space between drivers when moving about in a rush hour situation.
(4) Frequently, in presentations, I will say “in the end, it all comes down to the math.” While this is true, it lacks the motivation that drives the derivatives.
With today’s control algorithms we no longer deal with systems, rather we work with systems of systems. (For a comparison the refrigerator I purchased two months ago has a more advanced ECU then my first car)(1). Because of this, we need to consider fail-safe systems to protect against emergent properties.
Emergent properties can be defined as “a phenomenon whereby larger
entities arise through interactions among smaller or simpler entities such
that the larger entities exhibit properties, the smaller/simpler entities do not
Until a solution to the halting problem is found the best we can do is the creation of fail-safe systems. Fail-safe systems are characterized as
Inherently responding in a way that will cause no or minimal
harm to other equipment, the environment or to people.
My safe failures…
Starting off in the automotive industry there was a saying “a critical failure is too much energy; everything else is a fault.” Think of a vehicle oscillating out due to bad shocks or an engine overheating due to a lack of coolant; examples of kinetic and thermal energy overabundance.
If the problem is too much energy that the solution, obviously(2), is to remove energy. However, let’s take the two cases outlined above.
First in the case of the bad shocks. The “obvious” method for removing energy is to slow the car down by applying the brakes to slow the vehicle down. In this case, hard braking can add energy into the system; therefore a slow controlled braking event is required.
Something know is wrong, what should we do?
First off how do you know something is wrong? The answer should be that you have a fault detection system in operation. The fault detection sits outside of your system of systems and monitors the system. To monitor the system, you need the following information
- What are the “standard” operational parameters of the system? E.g. maximum engine temperature, biggest “bounce” of your spring?
- How long can you be outside of the “normal”?
- Are there multiple ways you can be abnormal?
- How do you pull the energy out?
The Northstar engine
During my first job, I worked at the GM development center in Flint Michigan. The work focused on simulations of the Northstar engine and ABS braking systems. At the time a unique feature of the Northstar engine was the ability to selectively shut down cylinders during steady state cruise; this was shown to provide a 5 ~ 6% fuel economy savings.
At some point during the diagnostic phase of development a bright engineer, not me, realized that the ability to selectively shut down cylinders also provided a way to cool down an overheating engine.
As our system of systems become more complicated, the protections for those systems have the worrisome trend of becoming more complex. Ensuring that the fault monitoring systems are independent of each other and do not interject faults into the system will be a topic for a future post.
(3) Growing up in Michigan during the 80’s Flint and the metro Detroit area was well past their heydays; still was and is a living community.
This video blog looks at fault detection and error handling. The included images of State Machines show templates for how I generally model fault detection algorithms.
In this first example there are two things to note:
- Debounce protection: Returning from “move to fault” and “no fault.” The signal needs to fall below the trigger signal – a delta to the signal. This prevents “jitter” in the signal. (Green circle.)
- Temporal logic: The move to “in fault” only takes place after you have held the fault condition as true for a set period of time. (Orange circle and black circle.)
The next example is more complex; in this example, a single variable “engine temp” can result in two different error modes. “High Temp” or “Critical High Temp.” In reality, the pattern is a slight variation on the previous version however it shows how it can be expanded to more complex fault conditions.
In a previous post, I wrote about general best practices for version control. In today’s posting, I am going to cover using version controls in the Model-Based Design workflow. Note for this blog post, I will be using generic terms for version control practices.
Derived and source objects
A derived object is any file that is generated from a source object or process. Within a traditional C language development process this would include
- .lib /.o / .exe
- Coverage reports
- Test results
- Data files
Like a traditional text-based language approaches a Model-Based Design has both source and derived objects; in the case of Model-Based Design the design objects are one level of abstraction removed from the C source files (1).
Utilities and project files
Within the MBD development process, there are both project and utility files(2). Version control best practices map are influenced by the data and architectural best practices. We will look at how releases are defined within this context.
What is a release? When to release?
Within the Model-Based Design context, a release is defined as the set of data and models that
- Contains consistent interfaces: the interfaces between the parent and child models are correct and the data definitions are compatible(3)across all the models.
- Reached a significant milestone: the models and data capture a significant portion of the desired functionality(4).
- Is stable: the models and the data pass a significant(5) percentage of the tests associated with the model.
Keep in mind, releases should not be made on a temporal basis. E.g. having a release every two weeks is not a good practice unless the conditions above are meet (at least condition 1 and 3.) At the same time allowing a long time to go between releases can make it more difficult for developers to integrate into the final project.
Version control, models, data and the system architect
Best practices for model architecture recommend a multi-level (integration level/child level) systems. Within this structure, the controls engineers are responsible for the checking in/out of their models. Further, they are responsible for ensuring the data that is unique to their models is correct.
The role of the system architect is to ensure that the individual models integrate together. They do this through the following tasks
- Validate that the model’s functional interfaces are compatible
- Validate that the use of global data is consistent across models(6)
- Running system level tests
In the case where there is a conflict between the functional interfaces or the use of global data, the system architect is responsible for arbitrating those conflicts.
Version control best practice for Model-Based Design or any software development is, in the end, dependent on communication between the team working on the software. The larger the team is the more the need for communication increases. Both documentation of the intended process and linking development to requirements will improve this process. In the end, try not to be this group…
(5) The collection of the models and the data do not need to pass all tests, rather they should pass the tests that engineering judgment are considered significant at that point in time. However, it should be noted that best practice is that as release advance towards the final release the number of tests passing should increase.
Within a software development organization, whether for embedded code or a desktop application, there are distinct roles. They are the controls engineer, the system architect, and the quality engineer. Depending on the size of the development team some of these roles may be done by a single person.
Analysis versus testing
During the development phase of a project, the controls engineer should perform analysis tasks on the model. These analysis tasks enable the controls engineer to determine if the algorithm they are developing is functionally correct and is compliance with the requirements.
It is common for the analysis tasks to be performed in an informal fashion. It is common for engineers to simulate a model and then view the graphs of the outputs to determine if they have correctly implemented the algorithm.
The differentiating word in this description is informal. When comparing analysis with testing we see that testing (either verification or validation) requires a formalized and “locked down” framework. How then can the informal analysis be used during the formal testing?
Transitioning from analysis to testing
Ideally, the transition from informal analysis to functional testing would flow seamlessly. However, it is often the case that the work done in the analysis phase is thrown away in the transition to the testing phase. This is understandable in a non-MBD environment but with the single truth approach of MBD, the analysis results should not be thrown away. This is where the idea of “golden data” comes into use. Golden Data is a set of data, both inputs, and outputs that an experienced engineer verifies as meeting the requirements of the algorithm.
Enabling the use of golden data to create test cases
The easiest way to enable the use of golden data is to provide controls engineers with a simple interface in which they can provide the analysis data set and the information that transforms it into a testing data set.
Analysis data is transformed into test data by providing a method for “locking down” the results. To locked down data the controls engineer needs to provide the test engineer information on what is expected from the analysis data. This information could include the following types of golden data tests.
- Strict: The output data from testing must match the golden output data exactly. This is normally done for Boolean or integer outputs.
- Tolerance: The output data from testing must match the golden output data within some bounded tolerance. Tolerances can be absolute, or percentage. Note special care needs to be taken with data with values around zero for percentage based tolerances.
- Temporal: The output from testing must match the golden output data within some time duration. These tests can also include tolerance and strict conditions.
In addition to the type of golden data tests to run the controls engineers should include information on which requirements the test maps onto.
Formal tests in support of development
In the same way that golden data can support testing the formal testing can support the controls engineers by informing them of the constraints that the requirements place on their design. This can only be achieved if the tests are easy for the controls engineers to run.
What is “user-friendly?”
User-friendly interfaces for testing are defined by the following characteristics
- Data is accepted in “natural” format: Any formatting or interpolation of the data is performed by the testing environment.
- Test results are presented in “human readable” format: The results from the tests should be provided both in a summary format (pass/fail) and with detailed data, such as graphs and tabular data.
- Selection and execution of tests should be simple: Tests should be launchable from a user interface that provides a list of the valid tests and enables the running of tests in either single or batch modes.
- Test files should be automatically associated: The management of test data (inputs and results) should be handled by the test manager.
This blog post has described how information should be shared and how tests should be run. In an upcoming post, I will cover the basics of modular test design.