Tag: #kpi

Assessing your current state

Published on by Michael Burke2 Comments

During the exploration phase of adoption, the adoption team should have become familiar with the fundamentals of Model-Based Design e.g. model architecture, data dictionaries, testing, formal methods, report generation and so on. In that this blog is focused on groups adopting MBD it is a fair assumption that the current state is “introductory” (1) however the following sections can be used to identify where additional support is needed.(2)

Architecture

The following tasks should be understood…

shellmodelint

  • Encapsulation of models:  How to define each model in a format with defined interfaces and data.
  • Integration of models: How to integrate individual models into a larger integration model
  • Integration of existing code artifacts: How to integrate artifacts from the models into existing code artifacts and/or integrate code artifacts into the models

Data management

The following tasks should be understood…

boxbarimage5.jpg

  • Creation of data artifacts for use in the model: How to create data that the model can reference
  • Management of data artifacts: How to store and reference data artifacts in a scalable way.
  • Harmonizing data between model and existing code artifacts: How to reuse data between models and existing code artifacts

Testing

The following tasks should be understood…
exampleHarness

  • Creation of test harnesses: How to create a test harness that will exercise the model in a “stand alone” method.
  • Creation of data: How to create the data used by the test harness both for input and output evaluation.
  • Creation of test evaluation methods: Creation of methods for evaluating the outputs from tests

Supporting tools

The following tasks should be understood…

supportingTools

  • Basic use of version control: The users should understand which files need to be placed under version control.

Final thoughts

The obvious question arises “how do I move from “introductory” to “ready?”  There are three primary methods

  1. Training: There are multiple training courses (industry) out there allowing you to learn about Model-Based Design (university).
  2. Papers: Both academic and industry papers exist to help you learn about MBD.
  3. Outside help:  Outside help can come from either hiring people with MBD experience or hiring outside consultants.
  4. All of the above…

Footnotes

(1) Having an introductory level of knowledge of Model-Based Design methodologies is not the same thing as an introductory level of knowledge of software development.  Most groups that adopt MBD have a strong software development background.
(2) Most of the “Why is my state so [XXX]” I understand.  However, I don’t know why Pennsilvina and Connecticut got “haunted” as their “Why.”

KPI for initial project…

Published on by Michael Burke1 Comment

What will and what were you measuring?

During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured.  However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).

First a word of warning, metrics are useful but they rarely provide the full picture(2).  That being said there are metrics that can be monitored

Bugs found at stage X…

One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process.bugs However, one side effect is the observation “we find so many bugs when following an MBD process.(3)”  Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process.  To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…

cost = Severity * StageConstant

Test and requirements coverage

Model-Based Design allows users to track the coveragePolyspacerequirements and testing coverage through automated tools(4).  With respect to test coverage; there are two types of test coverage.  The first is requirements based test coverage; do tests exist to cover the requirements.  The second are formal metrics coverage such as MCDC coverage.

The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.

Integration and development time

The final “primary” metrics are the development and integration time.  The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases).  The integration time is a bit more interesting.

In an ideal workflow for Model-Based Design, there is integrationHandsan integration model that is created at the start of the development cycle.  Individual developers check their models against that integration model to ensure that there are no conflicts.  Hence in an ideal flow, the integration time should be near zero.

However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes.  Like the bug detection finding integration issues is done further upstream in an MBD process.  Again the metric should use a weighted value based on the stage of where the integration issue is found.

Final thoughts

This post covered what can be measured, not how to measure them; this will be covered in future posts.  Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.

Footnotes

(1) For this post, I am assuming that you do not currently have metrics in place.  If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development.  Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process.  While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.

Adoption Time Line: Exploration Phase Part II

Published on by Michael Burke1 Comment

Continuing from an earlier post we now look at how you set the objectives for the initial adoption phase.

We need the champions

Before we proceed in setting objectives we need to talk about resources.  There are 3 resources that are required for an adoption process to succeed; they are

  1. Champions:  Technical and managerial support for the adoption process.  Without active advocates change will not happen.
  2. Time:  The champions need to have time allocated to working on the process change.  Ideally the technical champions will have 100% of their effort allocated to the adoption of the new process.  When the resources are allocated at less then 80% the change is likely to fail.
  3. Experience:  The people working on the project need to understand the current workflow so they can address its short comings and speak to the people outside of the adoption group.

An earlier blog post from LinkedIn provides additional details.

Setting goals

Based on the information collected from the process adoption team the objectives for the initial adoption phase should be set.  While the specifics for any given organization will be different the following outline is a fairly standard point

  1. Prior to start of initial adoption phase
    1. Allocate resources to the process adoption team in support of project
    2. Process adoption team completes identified required training
    3. Review reference materials to understand current industry best practices
  2. By completion  of initial adoption phase (1)
    1. Technical 
      1. Understand how artifacts from models integrate with existing artifacts
      2. Establish baseline testing activities
      3. Implement version control for new modeling artifacts
      4. Identify initial model and data architecture
    2. Managerial
      1. Review methods for measuring model key performance indicators (KPIs)
      2. Review resources required during initial adoption phase (2)

Bounding the problem

A word of caution; model-based design offers multiple tools and methods as part of the development workflow.  A common pitfall when establishing any new process is to “overreach” utilizing multiple new tools all at once, the resulting diluting of attention introduces errors of misunderstanding and results in a slower adoption of the process.  In the initial adoption phase posts, I will discuss the normal building blocks for Model-Based Design.

Next post

The next series of posts will cover model architecture and data management.  These topics will help in understanding the next phases of the adoption and establishment processes.

Footnotes

(1) The term “adoption” reflects the fact that there are existing resources to guide companies in adopting workflows.  I always encourage people to leverage existing information rather than creating new workflows from whole cloth.  This is critically important when working in a safety critical environment.
(2) Identifying the resources required for future phases should be based on the KPI information gathered from the initial adoption phase.  It should also take into account the “cost of learning” associated with starting a new process.

Adoption Time Line: Exploration Phase Part I

Published on by Michael Burke4 Comments

In our last post we reviewed the high level adoption time line; in this post, we will cover the exploration phase.  The objectives of the exploration phase are to

  • Understand the factors that influence the time required for adoption
    and
  • Set deliverable for the initial adoption phase.  (covered in part II)

From the last post, the factors influencing the adoption timeline are

  • Current usage of modeling tools:  The level of current knowledge and usage of modeling tools will influence the initial training and evaluation required.
  • Current software development processes:  The software processes adopted as part of a Model-Based Design workflow will leverage or extend many of your current software development processes.  The more formalized the existing processes the simpler the transition.
  • Compliance with high integrity software development processes: For a subset of software development projects compliance with high integrity processes such as the DO178-C or ISO 262622 imposes verification and traceability requirements that other processes can omit.
  • Processes required for development:  The Model-Based Design environment offers multiple tools for the development of software.  Determination of which tools and processes are required is done as part of the initial adoption phase.

The first step in exploration phase is to identify the stakeholders in the change process. and to understand the current development process both what works well and what needs improvement.  threeps

 

 

Role Deliverable
Controls Engineer Outlines current controls development workflow and roadblocks.
Software Engineer Outlines current software development process and method of integrating with the controls software
Project manager Outline current Key Performance Indicators (KPI)

Questions you should ask

  1. How do you currently connect requirements to
    1. Controls deliverable
    2. Testing
    3. Bug tracking
  2. What is your controls development process, waterfall, scrum,  design V?
  3. What are your current software tools
    1. Version control
    2. Bug tracking
    3. Test infrastructure
    4. Requirement tracking
  4. How is the software handed off between
    1. Controls and software engineers
    2. Controls and  test engineers
  5. What is your current testing process
  6. Where have most bugs entered into your current development process?

In the next post

In the next post, I will outline what questions you should ask moving forward.