Tag: #MBD

Bug tracking software

Published on by Michael Burke5 Comments

The first thing I need to clarify in this post is, what is and what is not a bug.

A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
unintended ways.

A software bug is not is

  • Incomplete features:  During the software development process features will be under development(1).  As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
  • Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software.  Again, as long as the lack of the feature does not introduce errors it is not considered a bug.

Incomplete features should already be tracked in the project planning timeline.  Desired features should be incorporated into the requirements document for the next generation of the project.

Severity of bugs

Not all bugs are created equal; defining the sevBeetle-Poster-720x479erity of bugs is necessary for prioritizing the correction of the bugs.  There are two common metrics for determining severity.  Frequency, how often does the bug occur.  Impact, when the bug occurs what happens to the program.  The following lists provide examples of how frequency and impact could be defined(2).

Frequency:

  • Infrequent:  Happens in less than 1% of the executions of the program in the normal work tasks.
  • Common: Happens for 1 ~ 5% of users in the normal course of work.
  • Prevalent: Occurs for 10% of users in their normal course of work.

Impact: 

  • Low:  Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
  • Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
  • High: Bugs that crash the program or create a loss of data for the user.

Once the frequency and impact have been determined the ranking of the bug can be defined.

Low Medium High
Infrequent Advised  Recommended Required
Common Advised Required Required
Prevalent Recommended Required Required

The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).

Bug fixing workflow

There are multiple commercially available tools for bug tracking.  A basic workflow should include the following tasks and events.

bugFix

  • Bug detection:  The bug is found either through use or captured in an existing test case
  • Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
  • Assignment:  The bug is assigned to a software engineer for resolution
  • Creation of test / validation of solution: If a test case does not already exist for the bug it should be created.  The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.

Final thoughts

Bug tracking and resolution is a problem common to all software development workflows.  The process for resolving these issues is the same for Model-Based Development as in traditional C development environments.  The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.

Footnotes

(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project.  See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development.  Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics.  Without the entry there is no method for determining the efficiency of the overall process.

What is a control algorithm?

Published on by Michael BurkeLeave a comment

The question, dreaded or loved that all engineers face is, “Tell me, what do you do?” As a controls engineer, I fall back to the following example.

Me: Imagine you are driving down the highway and you want to pass someone.  What do you do?
Imaginary person: Well I would shift over to the left lane and speed up.
Me: And if they started to go faster as you tried to pass them?
Imaginary person: Well I guess I would speed up some more.
Me: And after you pass them, what then?
Imaginary person:  Well I would move back over and slow back to my initial speed(1).

This simple example serves as a starting point for explaining the fundamentals of control algorithms.

  • The “objective”:  This is the thing we want to control.  In this example, it is the speed of our vehicle.passing-right
  • The input: How we affect the thing we want to control; for an automobile it is by depressing the acceleration pedal that the vehicle is commanded to go faster.
  • The feedback: The measurement of how close we are to our desired objective; e.g. are we going fast enough to pass the other car(2).

Everything is Newton’s method…

From the starting point of the automobile example, we can explain that the goal of a control algorithm is to drive the difference between the desired and the actual values to zero.  We can expand the driving example to drive our point home.newtons-method-example-graph

  • Overshoot:  When you stamp on the gas and end up going 80 instead of the 75 you needed to pass…
  • Fault detection: (Manual drivers only) when you try to hit 80 but are still in 2nd gear…
  • Adaptive controls: You drive more carefully in downtown Boston(3) than on a rural highway…

Final thoughts

As you could tell, this post is intended to be more light-hearted; however, I do find that thinking about controls problems in a non-technical relatable fashion helps me understand what I am talking about.

“It begins as an idea,
it ends with math(4)“

Footnotes

(1) We know this is an imaginary person since they said they slowed back down after passing.
(2)In this example we actually have an indirect measurement, are we passing the car not the actual speed of the vehicle.
(3) At least I hope you allow more space between drivers when moving about in a rush hour situation.
(4) Frequently, in presentations, I will say “in the end, it all comes down to the math.”  While this is true, it lacks the motivation that drives the derivatives.

Video Blog: Fault detection

Published on by Michael Burke2 Comments

This video blog looks at fault detection and error handling.  The included images of State Machines show templates for how I generally model fault detection algorithms.

In this first example there are two things to note:

  1. Debounce protection:  Returning from “move to fault” and “no fault.”  The signal needs to fall below the trigger signal – a delta to the signal.  This prevents “jitter” in the signal.  (Green circle.)
  2. Temporal logic:  The move to “in fault” only takes place after you have held the fault condition as true for a set period of time. (Orange circle and black circle.)

faultPattern

The next example is more complex; in this example, a single variable “engine temp” can result in two different error modes.  “High Temp” or “Critical High Temp.”  In reality, the pattern is a slight variation on the previous version however it shows how it can be expanded to more complex fault conditions.

criticalTemp

Model-Based Design and version control

Published on by Michael Burke4 Comments

In a previous post, I wrote about general best practices for version control.  In today’s posting, I am going to cover using version controls in the Model-Based Design workflow. Note for this blog post, I will be using generic terms for version control practices.

Derived and source objects

A derived object is any file that is generated from a source object or process.  Within a traditional C language development process this would include

  • .lib /.o / .exe
  • Coverage reports
  • Test results
  • Data files
  • …

Like a traditional text-based language approaches a Model-Based Design has both source and derived objects; in the case of Model-Based Design the design objects are one level of abstraction removed from the C source files (1).

automated-test-suite-generation-for-timecontinuous-simulink-models-4-638Utilities and project files

Within the MBD development process, there are both project and utility files(2).  Version control best practices map are influenced by the data and architectural best practices.  We will look at how releases are defined within this context.

What is a release?  When to release?

Within the Model-Based Design context, a release is defined as the set of data and models that

  1. Contains consistent interfaces: the interfaces between the parent and child models are correct and the data definitions are compatible(3)across all the models.
  2. Reached a significant milestone: the models and data capture a significant portion of the desired functionality(4).
  3. Is stable: the models and the data pass a significant(5) percentage of the tests associated with the model.

Keep in mind, releases should not be made on a temporal basis.  E.g. having a release every two weeks is not a good practice unless the conditions above are meet (at least condition 1 and 3.)  At the same time allowing a long time to go between releases can make it more difficult for developers to integrate into the final project.

Version control, models, data and the system architect

Best practices for model architecture recommend a multi-level (integration level/child level) systems.  Within this structure, the controls engineers are responsible for the checking in/out of their models.  Further, they are responsible for ensuring the data that is unique to their models is correct.

The role of the system architect is to ensure that the individual models integrate together.  They do this through the following tasks

  1. Validate that the model’s functional interfaces are compatible
  2. Validate that the use of global data is consistent across models(6)
  3. Running system level tests

In the case where there is a conflict between the functional interfaces or the use of global data, the system architect is responsible for arbitrating those conflicts.

releaseflow

Summary thoughts

Version control best practice for Model-Based Design or any software development is, in the end, dependent on communication between the team working on the software. The larger the team is the more the need for communication increases.  Both documentation of the intended process and linking development to requirements will improve this process. In the end, try not to be this group…

git

Footnotes

(1) For Model-Based Design, the “source files” could be C / C++ / VHDL / PLC or any other textual language.

(2) Utility files should also be placed under version control.

(3) For any given release the data may not be correct, e.g. having the final values as determined through requirements and testing; hence the term “compatible.”

(4) Note in some cases a release may be a ‘branch’ or experimental branch which may not be part of the final release of the software.

(5) The collection of the models and the data do not need to pass all tests, rather they should pass the tests that engineering judgment are considered significant at that point in time.  However, it should be noted that best practice is that as release advance towards the final release the number of tests passing should increase.

(6) Globally used data includes parameters and variables as well as structure definitions and enumerated data types.  As with all software development use of global data should be minimized.

User-friendly testing environments: Analysis and testing

Published on by Michael Burke2 Comments

Within a software development organization, whether for embedded code or a desktop application, there are distinct roles.  They are the controls engineer, the system architect, and the quality engineer.  Depending on the size of the development team some of these roles may be done by a single person.

Analysis versus testing

During the development phase of a project, the controls engineer should perform analysis tasks on the model.  These analysis tasks enable the controls engineer to determine if the algorithm they are developing is functionally correct and is compliance with the requirements.

It is common for the analysis tasks to be performed in an informal fashion.  It is common for engineers to simulate a model and then view the graphs of the outputs to determine if they have correctly implemented the algorithm.

The differentiating word in this description is informal.  When comparing analysis with testing we see that testing (either verification or validation) requires a formalized and “locked down” framework.  How then can the informal analysis be used during the formal testing?

Transitioning from analysis to testing

Ideally, the transition from informal analysis to functional combined-testing-analysistesting would flow seamlessly.   However, it is often the case that the work done in the analysis phase is thrown away in the transition to the testing phase.   This is understandable in a non-MBD environment but with the single truth approach of MBD, the analysis results should not be thrown away.  This is where the idea of “golden data” comes into use.  Golden Data is a set of data, both inputs, and outputs that an experienced engineer verifies as meeting the requirements of the algorithm.

Enabling the use of golden data to create test cases

who-moved-my-data-why-tracking-changes-and-sources-of-data-is-critical-to-your-data-lake-success-by-russ-savage-cask-3-638The easiest way to enable the use of golden data is to provide controls engineers with a simple interface in which they can provide the analysis data set and the information that transforms it into a testing data set.

Analysis data is transformed into test data by providing a method for “locking down” the results.  To locked down data the controls engineer needs to provide the test engineer information on what is expected from the analysis data.  This information could include the following types of golden data tests.

  • Strict:  The output data from testing must match the golden output data exactly.  This is normally done for Boolean or integer outputs.
  • Tolerance:  The output data from testing must match the golden output data within some bounded tolerance.  Tolerances can be absolute, or percentage.  Note special care needs to be taken with data with values around zero for percentage based tolerances.
  • Temporal: The output from testing must match the golden output data within some time duration.  These tests can also include tolerance and strict conditions.

In addition to the type of golden data tests to run the controls engineers should include information on which requirements the test maps onto.

Formal tests in support of development

In the same way that golden data can support testing the formal testing can support the controls engineers by informing them of the constraints that the requirements place on their design.  This can only be achieved if the tests are easy for the controls engineers to run.

What is “user-friendly?”

User-friendly interfaces for testing are defined by the following characteristics

  1. Data is accepted in “natural” format:  Any formatting or interpolation of the data is performed by the testing environment.
  2. Test results are presented in “human readable” format:  The results from the tests should be provided both in a summary format (pass/fail) and with detailed data, such as graphs and tabular data.
  3. Selection and execution of tests should be simple: Tests should be launchable from a user interface that provides a list of the valid tests and enables the running of tests in either single or batch modes.
  4. Test files should be automatically associated:  The management of test data (inputs and results) should be handled by the test manager.

friendly

Final thoughts

This blog post has described how information should be shared and how tests should be run.  In an upcoming post, I will cover the basics of modular test design.

 

 

If / elseif/ else: Why didn’t you ask me that in the first place?

Published on by Michael BurkeLeave a comment

Conditional logic, such as if/else, switch/case constructs, truth tables and state machines are common to most programming languages.  In today’s blog, there are two aspects of conditional logic that I want to address in today’s post,  independence, efficiency versus clarity.

Independence

Let’s compare two simple examples, in the first we have a single variable “A”, in the second we have three variables “A”, “B” and “C”

iflogicelseifstartingpoint

If we consider the first case the order of the comparisons (A==1) and (A==2) can be changed with no impact on the functional behavior of the code.  This is because the two conditions are mutually exclusive (1).  However in the second case, without formal analysis,  we cannot say if the order affects the functional behavior.  That is to say, if both A, B, and C can be true at the same time then the order of evaluation impacts the resulting functional behavior (2).

Frequently people are not aware of this possibility and as a have difficulty in evaluating the functional behavior of their code(3).

Efficiency versus clarity

For most people, the standard gear shift in cars is well known, with the order of states PRNDL as a standard (Park,  Reverse, Neutral, Drive, Low).  Therefore you commonly see conditional logic (or state logic) that represents the PRNDL like this…

prndl

However, from an efficiency point of view, this is poorly defined.  The majority of the time the transitions you will see are from park to reverse, park to drive, drive to park or reverse to park.  This would suggest the following organization

prndl_eff.

Why is this more efficient?  Simply put it reduces the average number of comparison operations required to reach the desired end state.

Compound conditions

The previous example was fairly straight forward; there is a single variable with independent states.  It is more common to find compound conditions defining the if/then/else logic.  Let’s look at our friends, A, B and C again.

compoundif

This first example is a classic compound if statement.  In this example, the logic is fairly straight forward.

deeplogicSo which of these is more readable?  This is a difficult question to answer.  In this case, the indented format of the code makes the binary nature of the if/else-if/else conditions clear.  However, if I introduced more complex evaluations, non-binary operations, then it would perhaps a hybrid of the two formats would be preferable.

Rules of thumb

  1. Keep the number of comparisons on a single line 3 or less:  Exceeding 3 comparisons per line make validation of the coverage of the case difficult.
  2. Keep the depth of if/if/if trees to 3 or less:  Traceability back to the top of the tree decreases at depths greater than 3.
  3. Place the binary comparisons at the top level of your if/if/if trees:   Placing the binary operations at the top of the tree reduces the overall number of if/else required.

Final thoughts

Formatting and density of information are frequently “hot topics” of debate.  The rules of thumb listed above should not be considered a final verdict but rather a starting point for discussion.  As always I welcome all comments.

Footnotes

(1) Note this may not be true if you move to a quantum computer where a single bit could maintain multiple states.  Until then you are fine this statement holds true.

(2) The second screen shot has a common error; lack of documentation.  The code should be commented to state if the code can be reordered.

(3)The common scenario has a person perplexed why they do not call “callBsFunction” when B is true.  Not realizing that it is gated by the “A” is true if statement.

Testing is software

Published on by Michael Burke5 Comments

This blog is a re-posting of early work from linkedin; I will be re-posting this week while I am at the Software Design for Medical Devices Europe conference in Munich.

Enabling the adoption of Model-Based Design

Test early, test often, test against requirements and test using formal methods. This is the mantra that developers (hopefully) hear. But what does it mean in practice? How do you produce effective and maintainable tests? I will argue that the first step is to think of test development in the same light as software development. Good testing infrastructure has requirements, is built from reusable components and written in a clear fashion to facilitate extensions and debugging efforts.

Why should you care?

In my 20+ years working in software, 2/3 of it in a consultative role, the most common problem I am called in to work on is mushroom code(1). Mushroom code is the end result of unstructured development, new algorithms are added on top of existing algorithms with little understanding of what it is feeding on. The result is an organic mess that is hard to sort out. This is prevalent in algorithmic development and even more common in testing which is often done “late and under the gun”

Testing components

A fully developed testing infrastructure consists of 5 components, a manager, execution methods, harnesses, reporting methods, and evaluation methods.

1.) Evaluation methods: use the data created through the execution of the test to determine the pass / fail / percentage complete status of the test:

Example a.) A MCDC test the evaluation would determine the percentage of conditions taken

Example b.) A regression test could compare the output values of between the baseline version of the code and the current release.

2.) Reporting methods: take the data from the evaluation methods and generate both human readable and history reports. The history reports are used to track overall trends in the software development process.(2)

3.) Harness: the harness provides a method for calling the unit under test (UUT) without modifying the UUT. Note test harnesses facilitate black box testing, e.g. the internal states of the unit under test are not known. However if internal states of the UUT are outputs at the root level of the model then white box testing can be done using the unit under test.(3)

4.) Execution methods: is how the test is run. This could be the simulation of a Simulink model, the execution of a .exe file, static testing (as with Polyspace) or the Real-Time execution (4)of the code.

As the name implies there is more than one “execution method.” They should be developed as a general class that allows the same method (simulation) to be applied to multiple harnesses. Each instance of a execution method applied to a harness is considered a test case.

5.) Test manager: is were all of these components come together. The test manager

  • Holds a list of the test cases
  • Automates the loading of associated test data
  • Triggers the execution of the test
  • Triggers the evaluation of the results
  • Triggers the generate of the test report

Sadly it will not yet fetch you a cold beverage.

Notes

1.) Mushroom code and spaghetti code are similar that they develop due to a lack of planning. Spaghetti code is characterized with convoluted calling structures; mushroom code is accumulation of code on top of code.

2.) An interesting list of what should go into a report can be found here.

3.) Any model can be turned into white box testing if global data is used. However the use of global data potential introduces additional failure cases.

4.) Yes, this blog retreads the work from 6 months ago, however it is good to review these issues.

Projects of interest (II): Listening for success

Published on by Michael Burke2 Comments

What is success? How do you define it, how do you measure it?  With software projects, it is easy to say when a project is complete but a complete project is not always a successful project.

So hear, at an abstract level, is my definition of a successful project.  The project…

  • solves the underlying (“big-picture”) requirements:  It is possible, even common, in the translating the initial (or user) requirements into derived requirements that the “big-picture” objectives are lost.  You see this reflected in tools and products that are functionally correct but more difficult to use or fail to provide the experience the user wants.
  • informs future work: A successful project can inform future work in two ways.  First directly, the work done on one project may be reused in subsequent projects.  The second is through acquisition of new knowledge(1) .
  • mentors junior people on the project: Every project is an opportunity for junior people to develop new skills and deeper understanding of the what they are working on(2) .

Background

17 years ago, on my first project as very green consultant,
I made the mistake of doing exactly what the green-consultantscustomer asked me to do.  Their request was to help them automate the routing of signals around a complex multi-level model.

I did what they asked, I learned a lot; efficient recursive programming, how to handle complex regular expressions, error handling for ill-defined systems.  The customer received a tool that did exactly what they asked for and they used it for the next 3 years.

So how was this project completed yet not successful?  First I didn’t step back to ask “what do they need?”  The customer thought they needed a way to route signals; in truth they needed a better model architecture.  The reason that they stopped using the tool 3 years later is that they realized this for themselves and developed a better model decomposition.

The second way in which this project failed is that it did not inform future work.  By its’ very nature it was a dead end tool; keeping them trapped  in an inefficient model architecture.  While I learned things that information was not applicable for my customer.

How to start succeeding?

Between the title of this post and my measures of success the answer should be clear.  At the start of my engagement I should have talked with and listened to my clients; that would have lead to the understanding that their architecture was in poor shape, I would have understood their underlying(3)  requirements.

student_success

Once you have the true objectives in mind make sure to review them periodically to ensure that the project has not drifted away from them.  Think about how the current project can inform future work, either directly through reuse or through education.  If it is for reuse budget the time in development to enable future reuse(4).

Footnotes

(1) There is a practical balance to be struck when learning new things on a project.  The time spent on learning new methods / tools should not slow down the progress of the project.  Ideally the knowledge would be gained organically while working on the project.

(2) Mentorship on projects is often informal; even basic act of discussing what and why design decisions have been made with junior colleagues will aid in their development.

(3) I am using “underlying” and “base” requirements to refer to the “big-picture” requirements from which all others are derived.  Given that the term for these big-picture requirements vary from field to field I hope that this will still be clear.

(4) Enabling reuse requires additional design and testing time.  A general rule of thumb is  to allocate an additional 10% ~15% of the development time.  I will write more about reuse in a future blog post.

Video Blog: Clarity in communication

Published on by Michael Burke2 Comments

Clarity of communication is key to any process; this short video covers a few thoughts on the importance of clarity in software development.

Selecting the initial project

Published on by Michael Burke1 Comment

When setting the scope of the initial project it is critical to remember the objectives of the initial phase as outlined in earlier posts.  They are:

  1. Understand how artifacts from models integrate with existing artifacts
  2. Establish baseline testing activities
  3. Implement version control for new modeling artifacts
  4. Identify initial model and data architecture

The motivation behind these objectives is to determine how the models fit into your overall process.  To do this a system of appropriate complexity needs to be selected.

Integration

Within the software community the term “spaghetti code”(1) is used to describe software that is poorly partitioned lacking in well-defined interfaces.  puzzleAs the name implies integration with spaghetti code is difficult to break apart and difficult to integrate in a “clean” fashion.

For the first project a section of code should be selected with a reasonably clean interface.  The primary thing to avoid is a section of code that is heavily dependent on global data.  For the initial system, you are almost always integrating the new component into the existing system.  You should target components with a limited number of I/O and no or limited dependence on global data.

(Note: this section has assumed that you are working on a project with existing code that must be integrated with.)

Questions to ask

Will the model need to…

  1. call existing supporting functions?
  2. access global data?
  3. have context dependent execution?

The objective is to select a model that minimizes those requirements.

V&V

The verification(2) and validation tasks(3) play a key part in determining the scope of the initial project.  A good module would include

  1. Modal systems:  State charts with event based logic; allows for verification of context based execution.
  2. Conditional execution: If/then/else logic and enabled / disabled subsystems; allows for testing of model coverage
  3. Dynamic systems: Requires closed loop simulation with stimulus; validates the response characteristics of the system.

11519783-vector-check-marks-stock-vector-check-box-tick

For each of these three items the expected outcome should be fairly well know; the objective is not to find major errors, though this sometimes happens during initial projects, but to understand how you will do these activities.

Questions to ask

Will the model…

  1. produce measurable outputs?
  2. have multiple modes of operation?
  3. have dynamic responses?

The objective is to select a model that maximizes these requirements.

Version control

While one of the objectives of the initial adoption is to understand the artifacts that will be placed under version control this objective has little impact on the setting of the scope of the system.  Version control will be covered in a later post.  For now you can read one of my earlier linked in blog posts on version control.

Model and data architecture

The selected component should enable you to validate your selected architecture as outlined in the model architecture and data sections of this blog.architecture

Questions to ask

Will the model need to have…

  1. parameterizable data calculations.
  2. a functional interface definition.
  3. conditional execution interfaces.

The objective is to select a model that maximizes these requirements.

Complexity

The final metric to consider is algorithmic complexity.  While Model-Based Design allows people to develop more complex algorithms, for the initial project an algorithm that is well understood should be selected.  The objective is to examine how models fit into your development workflow, not to validate the capabilities of the model(4).

Questions to ask

The algorithm should be

  1. an existing or extension of an existing algorithm
  2. part of the project that will first deploy the Model-Based Design process.

The objective is to select a model that maximizes these requirements.

Footnotes

(1) I prefer the term “mushroom code” as it better explains how this code comes about.  It is generally code that has grown on top of well written code but due to organic decay is now difficult to take apart.

(2) Verification is intended to check that a product meets a set of design specifications

(3) Validation is intended to ensure a product, that meets the operational needs of the user.

(4) The capabilities of the modeling environment should be assessed; however this is a separate task from developing your Model-Based Design workflow.

(5) The header image is a reference to the song from “The Sound of Music: Do-Re-Mi ” which provides the advice on where to start.