This post is the fifth in a series of 8 video blogs, walking through the fundamentals of Model-Based Design. When taken as a whole, these videos, provides the foundation stones for understanding and implementing Model-Based Design workflows. I will be using a simple Home A/C system for my example; however the principals apply to everything from Alpine ski sharpeners to Zambonide-icing.(1)
Today we will look at three V&V activities: guideline compliance, coverage and requirements based tests.
This post is the fourth in a series of 8 video blogs, walking through the fundamentals of Model-Based Design. When taken as a whole, these videos provide the foundation stones for understanding and implementing Model-Based Design workflows. I will be using a simple Home A/C system for my example; however the principals apply to everything from Active suspensions to Zonal control.(1)
With this post I cover the basics of data management, both for the model and configuration settings.
I like a good cup of coffee, while my wife Deborah is partial to tea and we both enjoy sharing a cup of hot cocoa on cold winter nights. (1) Recently we needed to buy a new kettle for our hot brews, so we purchased one that had a thermometer built in. This enabled brewing at the proper temperature, but we couldn’t tell what difference it was really making.
In the field, or in the model?
For experimental data it is common to “over log”; that is, to log as many data channels as your system can handle. This ensures if there is an unexpected event(2) you have the best chance of capturing and understanding the event. By contrast in simulation, the environment(3) is controlled and repeatable so unknowns are of low probability.(4) This means that “over logging” just slows down the simulation time.
How to determine what to log?
In the ideal world you are able to understand the system based off of first principles physics.(5) However, it is often the case that the system as a whole has too many interconnected models that writing out the full system of equations cannot be realistically performed. In that case how do you determine what to log?
The approach I recommend in this case is “self and nearest neighbors”. In other words if you cannot determine the full set of equations that define your whole system, break down the system into components (perhaps at the model reference or lower level) and determine what are the inputs and outputs of those systems. Take the inputs of your Unit Under Test (UUT) and the units directly connected to the UUT and use that to determine what to measure.
Back to coffee
I’ve started experimenting with coffee (okay, not as rigorously as in the milk first/tea first tea experiments), to determine the factors providing the optimal cup? There are 4 primary factors in the outcome of the cup of coffee.
Water temperature, Rate of extraction, coffee dose to water amount, coffee quality
The question then is, what is the relative weight to assign to each variable
Through a few simple experiments I learned my personal weights heavily lean towards β(3) and β(4) e.g. the temperature effect was minimal. (6) In the same way when designing models, think twice before measuring once. (7)
Footnotes
Perhaps one day we will buy this chocolate teapot for our hot cocoa
The “unexpected” is what you most want to capture; expected data can often be calculated, it is when the dice role snake eyes that you learn the most.
As an interesting side note, I often make 2 “plant” models. One that models the real world (the environment) and one that models the device I am controlling (e.g. the road and the car, air and an airplane, human veins and the I.V. system).
Unlike the real world where unknowns are random events, the unknowns in simulation arise from modeling errors, and when that occurs, adding in additional logging is important.
I was amused that the image of the book cover read “Note: this is not the actual book cover”! The use of a classical cover for fundamental physics seemed spot-on.
There is a side benefit to brewing at the correct temperature, fewer cases of “wow that’s hot” on the first sip of coffee
Unless you are cutting wood in which case it is design once; check your design and measure twice, cut once.
This post is the third in a series of 8 video blogs walking through the fundamentals of Model-Based Design. When taken as a whole, these videos provide the foundation stones for understanding and implementing Model-Based Design workflows. I will be using a simple Home A/C system for my example; however the principles apply to everything from Air filters to Zoetrope spin rate regulators.(1)
As a kid I made a number of “flip books“; I remember trying to use an old coffee can to make a Zoetrope, it was of our pet hamster using its drinking bottle. It was riveting…
This post is the second in a series of 8 video blogs walking through the fundamentals of Model-Based Design. When taken as a whole, these videos provide the foundation stones for understanding and implementing Model-Based Design workflows. I will be using a simple Home A/C system for my example; however the principles apply to everything from Arch welders to zygodactyl pickup systems.(1)
In this example I am laying out the system level architecture; in future posts I will go into the types of analyses that can be performed in this environment.
Legacy code, that is to say code from prior to your Model-Based Design environment can either act like an eagle allowing you to soar higher, or an albatross, a weight around your neck. Like the ships crew, think twice on your code and what you shoot for.
Migrate or encapsulate?
The decision on how to treat existing code is predicated on a series of questions.
In the current form, is it performing all the required functions? If the code (as is) is complete, then consider calling it in the “as is” fashion.
Dose the code need to be expanded or changed? If major changes are anticipated then migration to the Model-Based Design environment should be considered.
Is the code encapsulated? If the code is not well encapsulated then calling it from an external environment will difficult. The existing code should be refactored into smaller portions and then re-evaluated.
Dose the code perform functions not well suited for the Model-Based Design environment? There are some functions, such as device drivers that are best written in textual languages.
En-cap-so-early…
If the decision is made to bring existing legacy code into the Model-Based Design environment then efforts should be made to do this as early in the development process as possible to facilitate the testing of the encapsulated code. There are 3 types of tests that should be performed.
Simulation based tests: Does the legacy code provide meaningful data during simulation?
Code generation: Is the call to the legacy code correct in the generated code?
Encapsulation: Is the legacy code fully encapsulated, or does it require additional legacy code? Often this is an iterative process to pair down to the minimum code.
Simulation time stubs
In some cases the legacy code is collecting data from outside of the system e.g., in the cases where the code is for hardware or communication. When testing the code there are two options, static and dynamic “stubbing.”
In a static stub scenario, the return value(s) from the legacy code are set to a valid value that the code could provide.
In the dynamic stubbing scenario a model of the external code is created to simulate the values that the code could provide. To fully exercise the system the simulated code should provide not only expected values but potential error values which the system could produce.
Making the call
There are multiple ways that legacy code can be called from within the Model-Based Design environment. My current recommendation is to use the coder.ceval methodology.
This post is the first in a series of 8 video blogs, walking through the fundamentals of Model-Based Design. When taken as a whole, these videos provide the foundation stones for understanding and implementing Model-Based Design workflows. I will be using a simple Home A/C system for my example, however the principles apply to everything from Airplanes to Zeppelins.(1)
When I start a new project I go through the following basic stages
Identify what I am trying to solve: Review the requirements and ensure I know what it is I need to do.
Review methods that I know about on how to solve it: Most problems have multiple solutions; review the pros and cons.
Select method that fits within project constraints: Based on project constraints and the pros/cons, select the method.
Implement: Do the work.(1)
Refine: Based on feedback and time left, refine the approach.(2)
Today I want to talk about the refinement stage: what it is, when, and how to do it.
Before you run a marathon, have a good running form
Refinement starts once the baseline version of the project is completed, that is to say you have
Met all the baseline requirements
Have unit tests in place for all the baseline requirements
Have completed integration into the larger system
Refining!
If you do it poorly you will pay penalties over and over(3) but done well, there are great benefits. So what should you refine? In design there are 4 things to optimize, they are: (4)
Clarity: Improving the design to make it easier to understand and maintain.
Speed: Reducing the total FLOPS for both mean and edge cases!
Memory usage: Reducing the RAM/ROM, for both mean and edge cases!
Re-usability: Create methods for reusing the design for multiple instances.
What is most important to your project?
Items 1 through 4 are often in competition; it is easier to go faster if you use more memory but clarity may suffer if you have multiple uses. So the question becomes, “what is most important to your project as a whole?” Take into account how often the code is called and determine its priority within the execution context.
Clarity
The first rule of clarity is documenting your model. Documentation should be done at a level that aids understanding. Putting comments next to each block (this is a table look-up for calculating the XXX based on YYY) provides too much information. Comment on the function of a group of blocks. The second rule is to use common patterns; do not have 6 different ways for performing the same operation.
Speed
Before you can go fast you need to know what is going slow. The first step in optimization then is to run a profiler on your system. There are two types of speed you want to examine; average and worst case. Depending on your overall system needs you may want to focus on one over the other. This can be done by implementing conditional code where a given path only executes under edge cases.
Memory
First off, elephants are a horrible storage medium; sure, you pay them in yummy peanuts but the retrieval process is a real three ring circus. Memory can be reduced in two primary methods
Reduce what you pass in: Look at the data you are passing into your function. Is all of it needed? Frequently in the development process we “overpack.”
Remove intermediate variables: This is less of an issue for automatic code generation. However, if you find yourself creating variables to monitor the code this will lead to memory bloat.
Reusability
Rather then write something again, I will reuse these posts on reuse… (Yes, I am reusing this joke)
Final thoughts
Remember as you refine your code, regression tests are your best friend. Consider adding performance tests as well if you are working on memory and speed issues. Follow this advice and “go for the gold!”
While doing the work may take the most time, steps 1-3 are what makes sure that it takes the least amount of time.
For projects that are well defined, the “refinement” may take place while in stage 4 (implementation). However, when doing something new I always make sure I have something working and solid before I start the refining stage.
I’m not saying that is why it is re-fine, e.g. a penalty paid but…
In computer programming this is known as “doing your bit”; if you “byte off more than you can chew.”
A recent article announced “Deep Learning Machine Beats Humans in IQ Test.” It is an interesting read but it raises a question on many minds these last few years; how do you test a device that is an unbounded black box?(1)
Testing DL/ML programs is a unique challenge. Unlike conventional software testing, the algorithm you are testing does not have a viewable code base (e.g. you can’t inspect for overflow, underflow, or pointers). So what do you test?
The first question is “what are you protecting against?” This is not the inverse of what you are trying to achieve. Let’s take a hypothetical case; you have developed a ML/DL system that controls a robot to give full body massages.(2)
The objective of the robot is to relive pain and tension in the body. What we are protecting against is damage to the body such as bruising or muscle tears.
Another way of thinking about this is the “law of unintended consequences.” Reading Asimov’s Robot stories you would see that the three laws were not enough to keep things from going off the rails. So what can we do?
Fencing the problem
In some cases the solution is simple; put an observer in place and fence off the dangerous areas. In our massaging robot example, the “fence” could include things such as
Maximum force
Maximum compression
Maximum speed
In some cases fencing isn’t a solution so how do we remain “en-guard”?
Black swans
There is no way to dance around this one; AI/ML/DL systems in the real world have to deal with real world problems. That is to say they need to “expect the unexpected.”(3)
AI/ML/DL systems work based on data models; any event that is outside of the data model is still responded to and interpreted as something that is known. Since it is not possible to know all the unknowns upfront the question becomes “are there meta-unknowns?” Can you come up with classes of things that you have not trained for, that you can give rules for how to respond?
Sticking with our massaging robot, someone with scar tissue from surgery may have a different underlying musculature layout. This is a different problem from someone who has a temporary strain that can be corrected.
Fail Safe
To “fail safe” we need to first know what “safe” is. Fail safe for an engine when you are parked would be to shut down whereas while driving it may be to reduce power. AI/ML/DL systems can employ scenario based fail safe approaches to determine what the correct action is when a black swan or fence condition occurs.
0.3048 Meter notes(4)
I’m describing this as an unbounded black box since the actual range of inputs cannot be defined as in a traditional system. For example, you could say a voltage signal goes from 0 to 42 volts, but how do you define the information encoded in an image when you don’t know what the system has determined to be salient features?
I’m not saying that sitting at a desk much of the day has made me interested in creating such a machine but…
For many years the “No one expects the Spanish Inquisition” was a fine joke; however given time, everyone expects the Spanish Inquisition and it is no longer an edge case.
1 foot is 0.3048 meters, hence these can be translated into “foot” notes.
Do you have a question about Model-Based Design? About Juggling, swimming, how Model-Based Design could make your Juggling Routines go Swimmingly? Here is your chance to ask me, and I will respond in an upcoming post!
While you are at it, sign up and get this blog by email.
Footnotes
If I included spam I would have questions like:
“Do you want to be a millionaire?” Yes, and a socially responsible one.
“Your car warranty is running out! Do you want to renew?” While they got the car right, it was one that my wife Deborah and I sold 20 years ago.
