Tag: #metrics

KPI for initial project…

Published on by Michael Burke1 Comment

What will and what were you measuring?

During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured.  However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).

First a word of warning, metrics are useful but they rarely provide the full picture(2).  That being said there are metrics that can be monitored

Bugs found at stage X…

One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process.bugs However, one side effect is the observation “we find so many bugs when following an MBD process.(3)”  Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process.  To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…

cost = Severity * StageConstant

Test and requirements coverage

Model-Based Design allows users to track the coveragePolyspacerequirements and testing coverage through automated tools(4).  With respect to test coverage; there are two types of test coverage.  The first is requirements based test coverage; do tests exist to cover the requirements.  The second are formal metrics coverage such as MCDC coverage.

The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.

Integration and development time

The final “primary” metrics are the development and integration time.  The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases).  The integration time is a bit more interesting.

In an ideal workflow for Model-Based Design, there is integrationHandsan integration model that is created at the start of the development cycle.  Individual developers check their models against that integration model to ensure that there are no conflicts.  Hence in an ideal flow, the integration time should be near zero.

However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes.  Like the bug detection finding integration issues is done further upstream in an MBD process.  Again the metric should use a weighted value based on the stage of where the integration issue is found.

Final thoughts

This post covered what can be measured, not how to measure them; this will be covered in future posts.  Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.

Footnotes

(1) For this post, I am assuming that you do not currently have metrics in place.  If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development.  Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process.  While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.

Bug tracking software

Published on by Michael Burke5 Comments

The first thing I need to clarify in this post is, what is and what is not a bug.

A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
unintended ways.

A software bug is not is

  • Incomplete features:  During the software development process features will be under development(1).  As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
  • Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software.  Again, as long as the lack of the feature does not introduce errors it is not considered a bug.

Incomplete features should already be tracked in the project planning timeline.  Desired features should be incorporated into the requirements document for the next generation of the project.

Severity of bugs

Not all bugs are created equal; defining the sevBeetle-Poster-720x479erity of bugs is necessary for prioritizing the correction of the bugs.  There are two common metrics for determining severity.  Frequency, how often does the bug occur.  Impact, when the bug occurs what happens to the program.  The following lists provide examples of how frequency and impact could be defined(2).

Frequency:

  • Infrequent:  Happens in less than 1% of the executions of the program in the normal work tasks.
  • Common: Happens for 1 ~ 5% of users in the normal course of work.
  • Prevalent: Occurs for 10% of users in their normal course of work.

Impact: 

  • Low:  Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
  • Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
  • High: Bugs that crash the program or create a loss of data for the user.

Once the frequency and impact have been determined the ranking of the bug can be defined.

Low Medium High
Infrequent Advised  Recommended Required
Common Advised Required Required
Prevalent Recommended Required Required

The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).

Bug fixing workflow

There are multiple commercially available tools for bug tracking.  A basic workflow should include the following tasks and events.

bugFix

  • Bug detection:  The bug is found either through use or captured in an existing test case
  • Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
  • Assignment:  The bug is assigned to a software engineer for resolution
  • Creation of test / validation of solution: If a test case does not already exist for the bug it should be created.  The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.

Final thoughts

Bug tracking and resolution is a problem common to all software development workflows.  The process for resolving these issues is the same for Model-Based Development as in traditional C development environments.  The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.

Footnotes

(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project.  See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development.  Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics.  Without the entry there is no method for determining the efficiency of the overall process.

Initial adoption: objectives and metrics

Published on by Michael BurkeLeave a comment

Objectives and metrics

Based on the information collected from the process adoption team the objectives for the initial adoption phase should be set.  While the specifics for any given organization will be different the following outline is a standard view.

1403204517-3-keys-measure-success-loyalty-program

  1. Technical 
    1. Complete 1 or 2 “trial” models
      1. Identify the initial model architecture
      2. Identify the initial data architecture
      3. Establish baseline analysis methods
      4. Establish baseline testing methods
    2. Understand how artifacts from models integrate with existing artifacts
    3. Implement version control for new modeling artifacts
  2. Managerial
    1. Review methods for measuring model key performance indicators (KPIs)
    2. Review resources required during initial adoption phase

The technical metrics

Completion of the trial models

In a future post we will examine how to select your trial model,  but for now let’s answer the question “what does it mean to complete a trial model? ”  This decomposes into the four tasks as outlined above.  The model and data architecture are covered in some depth in previous posts so let us talk about analysis and testing.

Within the Simulink domain, a fundamental aspect of a model is the ability to simulate the behavior of the plant or the control algorithm. analog-simulationThe simulation is used during the early stage of development to analyze the model to determine if the functional behavior of the model.  The developer performs elaborates the model until the behavior functionality matches the requirements.  This is verified through simulation.  Once the model meets the requirements the functionality can be “locked down” through the use of formal tests; again using simulation.

It is worth noting that some requirements will be met before others, they should be formally locked down under test as they are achieved.

Integration with existing artifacts

For most companies, unless they are starting from
integratesoftwarea clean sheet there will be existing software components that need to be integrated with those created by the Model-Based Design process.  There are three types of integration

  1. bringing existing software into the Model-Based Design framework
  2. Bringing Model-Based Design artifacts into the existing architecture.
  3. A combination of 1 and 2.

The topic of integration will be covered in greater detail in an upcoming post.  However, the fundamental guidelines for integration (in either direction) are the following.

  • Create software objects with well-defined interfaces (encapsulation)
  • Limit dependencies of the software objects on external objects
  • Minimize the use of “glue code”(1).

Version control of objects

Version control processes use tools to enable team-based development while maintaining multiple “release” or “branches”.  During the initial phase of the project which software objects will be placed under control and how a “release” will be defined should defined.  This initial definition will be refined as the Model-Based Design process is elaborated.  This blog will go into detail on this in a future post.  The basic questions are

  • Do you include derived objects in the version control software: .c, .h, test reports…
  • How do you arbitrate check in conflicts: How do engineers resolve conflicts in their model interfaces?
  • How do you share data / models across projects: What methodology will facilitate reuse of objects across multiple projects with minimal configuration impact?

Managerial metrics

The initial adoption phase by its nature will be slower than later phases as people are still learning new capabilities and processes.  57572027-gears-and-kpi-key-performance-ind-cator-mechanismThe primary objectives during this phase are

  1. Learn what bottlenecks with the new process.
  2. Understand existing issues uncovered by the transition
  3. Determine level of resources for the next stage

The discovery of “Objective 2’s”, existing issues, often surprises people.  The act of transitioning to a new process forces the evaluation of existing processes and, more often than not, illuminates existing shortcomings.  Extra care should be taken to ensure that the new process addresses those shortcomings.

In the next stage, the validation project, the team should expand beyond the initial “core” team. Ideally, a people from outside the initial project scope should be brought in to identify developmental pain points that did not exist in the “core group” processes.

Footnotes

(1): “Glue code” is a software object created solely for the connection of two other software objects.