Month: March 2017

Behind the M.A.A.B. Style Guide: Lessons in working with brilliant people

Published on by Michael BurkeLeave a comment

[This reprint of one of my linked in blog posts while I am on vacation]

For 8 years, between 2005 and 2013, I had the privilege of hosting the MathWorks Automotive Advisory Board (MAAB). In that role I helped to shape of version 2.0, 3.0 and 3.1 of the document. Over the course of those 8 years I enjoyed many conversations with engineers from GM, Ford, John Deere, Toyota, CAT to name but a few. Working with them helped to sharpen my listening and understanding skills; a benefit that I carry forward in all of my work.

Who were those masked people?

The average member of the MAAB Style Guide working group had a PhD, 20+ years of industry experience, and were the Model-Based Design subject matter expert within their company. Most importantly of all they were passionate about developing the field of Model-Based Design (a majority of the board members donated their time outside of their normal work duties.)

Bright, passionate used to being listened to…

When I took over the position the former host likened it to “herding cats.” So how to get them to work together?

1.) Understand your own bias: It is an extremely rare to take part in a conversation where you have no opinion. Be aware of your own thoughts and how they influence what you are hearing. Use that knowledge to be open to new understandings

2.) Actively listen to them: The first step in reaching consensus is understanding all points under discussion. As the “outside” person it is easier to hear all points of view

3.) Find the common ground: In most cases two (or more) parties will share common ground in their base line objectives.

Define the common ground referencing what you learned through active listening, and reach a consensus from the group for this baseline information.

4.) Explore the reasons behind the differences:

In my experience 80% of the time the differences in approach stemmed from a real world problem that the other person had yet to encounter.(1) Once the rationale behind the difference was understood then an informed response can be made

5.) Accept compromise: For the 20% of cases where there is no agreed upon position can be found (2) there is an appeal to necessity. Most experienced engineers understand project deadlines and if they have been listened to (3) are willing to accept a compromise on the work product.

We are all those masked people

In 20+ years of direct customer interactions, both technical and business focused, I have come to realize that everyone can be a ‘brilliant person.’ What you get out of an conversation is directly proportional to what you put into conversation.

Notes:

1.) The objective of the MAAB Style Guidelines are to provided recommendations to ensure error free models with a consistent style to aid in understanding.

2.) With the MAAB Style Guidelines there were cases where consensus could not be reach and both parties had valid points. This is where you see the “follow either option 1 or option 2” in the guidelines.

3.) Please note, I write, “have been listened to” not “feel like they have been listened to”. People can tell if you have actively listened and understood them.

What’s on your white board…

Published on by Michael BurkeLeave a comment

[This blog post is a “reprint” of one of my Linked in posts while I am on vacation]

In meetings or in your office the humble whiteboard(0) is a rapid prototyping environment through which we communicate ideas. Chances are if I have ever spoken with you in a room with a whiteboard, I have drawn this(1) representation of Model-Based Design system architecture.

As I drew I would have spoken about the following concepts

1.) The Green: model hierarchy:

a.) To support team based and unit testing development workflows create your models in a hierarchical fashion. Use model reference to encapsulate each level of the model.

b.) The “final” level of the hierarchy should be a model that is owned by a single developer and should be of a size and complexity that someone besides the developer (2) can understand it with an hour review.

2.) The Red: separation of hardware

a.) Calls to hardware functionality should be separated from the algorithmic code

b.) For simulation based testing create “I/O” which enables the

i.) Validate of interfaces

ii.) Stimulate the algorithmic model with “safe” values

c.)Within the hardware I/O modules convert the date from digital (counts, volts,…) to engineering units.

3.) The Blue: version control / software practices…

a.) Place the objects into version control (e.g. model, data, test files, reports…)

b.) Identify objects that can be reused across projects

c.) Requirement documents, test results, and utility functions should exist within the repository (3)

4.) The Black: Data and tests

a.) Create a data dictionary that exists separate from the models

i.) The data dictionary includes model and test data (test inputs and results)

ii.) The data dictionary includes model and build configuration information

b.) Create a generic test infrastructure for

i.) Running the tests

ii.) Creating test reports

iii.) Performing common test validation tasks

c.) Reports: use the testing infrastructure to create reports for status tracking and internal review

Cleaning the board

I would be interested in seeing what is on your white board. To start things off I have captured a few of my co-workers boards….

Information dense

Delta estimates

Project specific

Footnotes:

0.) I still prefer chalk boards to white boards(4) even though I see the advantages of white boards; I just like the tactile feel.

1.)   Honestly if you have seen me draw this in person it was not this neat and clean as the discussion would have included redrawn lines to incorporate the bits of your own process.

2.)   Always be ready for the developer of an algorithm to move on; 95% of the time I want clarity over cleverness in design.

3.)   The requirements documents don’t show up in this white board picture, this is to represent the reality of many engineering environments

4.) These footnotes are using C zero-based indexing.

Model-Based Design F.A.Q.

Published on by Michael BurkeLeave a comment

In a recent meeting, I was asked: “What are the common questions that people ask about Model-Based Design?”  With some reflection, I came up with these four questions and answers.

What are the benefits of adopting Model-Based Design?

The financial benefits?

This question, which generally comes from management is straight forward.  Do we see a financial benefit (ROI) from adopting?  If so how long before we see these benefits?  The answer, of course, depends on the starting point of the group, however, the following link provides a good overview on estimating ROI for the adoption process.  The timeline for realizing benefits is generally 6 months to start seeing the ROI.  Full adoption and full realization occur between 2 ~ 4 years after the start of the project.

The process benefits?

This is, more often than not, processa defensive question. There is, understandably, a reluctance to change existing processes if they are working.  Because of this, the answer addresses the common problem that existing traditional hand coding environments, errors introduced at the handoffs points between researchers to developers to test engineers to system engineers to…  As discussed in earlier blog posts, one of the fundamental aspects of Model-Based Design is the use of a model as the single development environment throughout the full development process.

I have a technical question…

How do I reuse all these existing…

For most companies, there is areuse significant investment in existing infrastructure.  Therefore, it is a high priority task to figure out how existing software objects.  First, as talked about in these blogs much of the existing software development infrastructure can and should be reused with some modification and extensions.

Second, there is the perennial engineering advice: “If it ain’t broke don’t fix it.”  One of the earliest tasks  in adopting Model-Based Design is identifying the parts of the existing software infrastructure that is

  1. Well partitioned and encapsulated
  2. Reflects and maps to the requirements
  3. Has good test coverage

Once those have been identified, leave them alone.  For other software objects, they can either be recreated or they can be incorporated into the Model-Based Design environment depending on their level of “completeness.”

How can it ever be as good as what Sally does?

Sally, or Bob, depending on the day I am speaking, is the local guru.  They are the person who produces the best code, tests, or systemsavant architecture following a traditional design practice.

The short answer that I give is this: no automation process, for a single task, can match a highly experienced developer.  However, there is a limit to how many Sallys/Bobs that exist in an organization and the automation software produces code that is on par with an average developer for single tasks.  Further, due to the optimizations that are possible for an automated methodology, it can find optimizations that humans have difficulty in finding.

There is another aspect to this; has anyone asked Bob and Sally if they would like to be working on higher level tasks?  Frequently these people could accomplish even more if they were not down in the “guts”

 

Assessing your current state

Published on by Michael Burke2 Comments

During the exploration phase of adoption, the adoption team should have become familiar with the fundamentals of Model-Based Design e.g. model architecture, data dictionaries, testing, formal methods, report generation and so on. In that this blog is focused on groups adopting MBD it is a fair assumption that the current state is “introductory” (1) however the following sections can be used to identify where additional support is needed.(2)

Architecture

The following tasks should be understood…

shellmodelint

  • Encapsulation of models:  How to define each model in a format with defined interfaces and data.
  • Integration of models: How to integrate individual models into a larger integration model
  • Integration of existing code artifacts: How to integrate artifacts from the models into existing code artifacts and/or integrate code artifacts into the models

Data management

The following tasks should be understood…

boxbarimage5.jpg

  • Creation of data artifacts for use in the model: How to create data that the model can reference
  • Management of data artifacts: How to store and reference data artifacts in a scalable way.
  • Harmonizing data between model and existing code artifacts: How to reuse data between models and existing code artifacts

Testing

The following tasks should be understood…
exampleHarness

  • Creation of test harnesses: How to create a test harness that will exercise the model in a “stand alone” method.
  • Creation of data: How to create the data used by the test harness both for input and output evaluation.
  • Creation of test evaluation methods: Creation of methods for evaluating the outputs from tests

Supporting tools

The following tasks should be understood…

supportingTools

  • Basic use of version control: The users should understand which files need to be placed under version control.

Final thoughts

The obvious question arises “how do I move from “introductory” to “ready?”  There are three primary methods

  1. Training: There are multiple training courses (industry) out there allowing you to learn about Model-Based Design (university).
  2. Papers: Both academic and industry papers exist to help you learn about MBD.
  3. Outside help:  Outside help can come from either hiring people with MBD experience or hiring outside consultants.
  4. All of the above…

Footnotes

(1) Having an introductory level of knowledge of Model-Based Design methodologies is not the same thing as an introductory level of knowledge of software development.  Most groups that adopt MBD have a strong software development background.
(2) Most of the “Why is my state so [XXX]” I understand.  However, I don’t know why Pennsilvina and Connecticut got “haunted” as their “Why.”

KPI for initial project…

Published on by Michael Burke1 Comment

What will and what were you measuring?

During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured.  However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).

First a word of warning, metrics are useful but they rarely provide the full picture(2).  That being said there are metrics that can be monitored

Bugs found at stage X…

One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process.bugs However, one side effect is the observation “we find so many bugs when following an MBD process.(3)”  Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process.  To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…

cost = Severity * StageConstant

Test and requirements coverage

Model-Based Design allows users to track the coveragePolyspacerequirements and testing coverage through automated tools(4).  With respect to test coverage; there are two types of test coverage.  The first is requirements based test coverage; do tests exist to cover the requirements.  The second are formal metrics coverage such as MCDC coverage.

The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.

Integration and development time

The final “primary” metrics are the development and integration time.  The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases).  The integration time is a bit more interesting.

In an ideal workflow for Model-Based Design, there is integrationHandsan integration model that is created at the start of the development cycle.  Individual developers check their models against that integration model to ensure that there are no conflicts.  Hence in an ideal flow, the integration time should be near zero.

However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes.  Like the bug detection finding integration issues is done further upstream in an MBD process.  Again the metric should use a weighted value based on the stage of where the integration issue is found.

Final thoughts

This post covered what can be measured, not how to measure them; this will be covered in future posts.  Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.

Footnotes

(1) For this post, I am assuming that you do not currently have metrics in place.  If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development.  Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process.  While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.

Model-Based Design foundational concepts

Published on by Michael BurkeLeave a comment

In previous blog posts, I have gone into some depth on testing, architecture and data management for models.  With this post, I will cover how these three activities for the foundation of any Model-Based Design process.

Foundations and core competencies

At later stages in the adoption of Model-Based Design crawl-walk-run-fly-300x191
processes task-specific groups will emerge (development, verification, and systems.)  However, at the start of the Model-BAsed Design process users from all groups need to determine the “common language” that will be used to develop their project (1).

Architecture and data sure, but why testing?

The identification of architecture and data as foundational concepts is generally well understood.modelcentricgifslow  Combined they define how people will develop the model through interfaces and clear communication.  So why testing in the trinity?  It returns to the core concept of Model-Based Design that models are at the center of the development process.  To ensure that the model can be used consistently through the development process they need to be “locked down”(2) with test cases.

Driven by this objective, the testing environment is designed at the start of the development process.  The requirements of the test environment should be addressed within the architectural and data infrastructure.  The good news is that best practices for the three “legs”(2) of the MBD stool are already in the well defined; it is a matter of honing them to your specific project and environment.

Final thoughts

This post is not intended to be technical; rather it is to remind us as we develop new processes to start out with the “best path forward” from the start.  In the section about the validation project, I will discuss the next round of tools that are commonly adopted.

Footnotes

(1) I may start using this graphic as my tag for MBD adoption.  Crawl = investigate.  Walk = initial.  Run = validated.  Flying = optimizing.
(2) Tests cases are elaborated as the model is developed.  The “lock down” is achieved through the use of a continuous build and test server.
(3) The metaphor of a tripod or stool can be overused.  But, to push it one last time, this is your stepping stool to the next round of MBD tools and processes.  Build it well and it forms a strong foundation.

Bug tracking software

Published on by Michael Burke5 Comments

The first thing I need to clarify in this post is, what is and what is not a bug.

A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
unintended ways.

A software bug is not is

  • Incomplete features:  During the software development process features will be under development(1).  As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
  • Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software.  Again, as long as the lack of the feature does not introduce errors it is not considered a bug.

Incomplete features should already be tracked in the project planning timeline.  Desired features should be incorporated into the requirements document for the next generation of the project.

Severity of bugs

Not all bugs are created equal; defining the sevBeetle-Poster-720x479erity of bugs is necessary for prioritizing the correction of the bugs.  There are two common metrics for determining severity.  Frequency, how often does the bug occur.  Impact, when the bug occurs what happens to the program.  The following lists provide examples of how frequency and impact could be defined(2).

Frequency:

  • Infrequent:  Happens in less than 1% of the executions of the program in the normal work tasks.
  • Common: Happens for 1 ~ 5% of users in the normal course of work.
  • Prevalent: Occurs for 10% of users in their normal course of work.

Impact: 

  • Low:  Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
  • Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
  • High: Bugs that crash the program or create a loss of data for the user.

Once the frequency and impact have been determined the ranking of the bug can be defined.

Low Medium High
Infrequent Advised  Recommended Required
Common Advised Required Required
Prevalent Recommended Required Required

The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).

Bug fixing workflow

There are multiple commercially available tools for bug tracking.  A basic workflow should include the following tasks and events.

bugFix

  • Bug detection:  The bug is found either through use or captured in an existing test case
  • Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
  • Assignment:  The bug is assigned to a software engineer for resolution
  • Creation of test / validation of solution: If a test case does not already exist for the bug it should be created.  The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.

Final thoughts

Bug tracking and resolution is a problem common to all software development workflows.  The process for resolving these issues is the same for Model-Based Development as in traditional C development environments.  The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.

Footnotes

(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project.  See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development.  Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics.  Without the entry there is no method for determining the efficiency of the overall process.

What is a control algorithm?

Published on by Michael BurkeLeave a comment

The question, dreaded or loved that all engineers face is, “Tell me, what do you do?” As a controls engineer, I fall back to the following example.

Me: Imagine you are driving down the highway and you want to pass someone.  What do you do?
Imaginary person: Well I would shift over to the left lane and speed up.
Me: And if they started to go faster as you tried to pass them?
Imaginary person: Well I guess I would speed up some more.
Me: And after you pass them, what then?
Imaginary person:  Well I would move back over and slow back to my initial speed(1).

This simple example serves as a starting point for explaining the fundamentals of control algorithms.

  • The “objective”:  This is the thing we want to control.  In this example, it is the speed of our vehicle.passing-right
  • The input: How we affect the thing we want to control; for an automobile it is by depressing the acceleration pedal that the vehicle is commanded to go faster.
  • The feedback: The measurement of how close we are to our desired objective; e.g. are we going fast enough to pass the other car(2).

Everything is Newton’s method…

From the starting point of the automobile example, we can explain that the goal of a control algorithm is to drive the difference between the desired and the actual values to zero.  We can expand the driving example to drive our point home.newtons-method-example-graph

  • Overshoot:  When you stamp on the gas and end up going 80 instead of the 75 you needed to pass…
  • Fault detection: (Manual drivers only) when you try to hit 80 but are still in 2nd gear…
  • Adaptive controls: You drive more carefully in downtown Boston(3) than on a rural highway…

Final thoughts

As you could tell, this post is intended to be more light-hearted; however, I do find that thinking about controls problems in a non-technical relatable fashion helps me understand what I am talking about.

“It begins as an idea,
it ends with math(4)“

Footnotes

(1) We know this is an imaginary person since they said they slowed back down after passing.
(2)In this example we actually have an indirect measurement, are we passing the car not the actual speed of the vehicle.
(3) At least I hope you allow more space between drivers when moving about in a rush hour situation.
(4) Frequently, in presentations, I will say “in the end, it all comes down to the math.”  While this is true, it lacks the motivation that drives the derivatives.