Tag: #process

KPI for initial project…

Published on by Michael Burke1 Comment

What will and what were you measuring?

During the initial phase of the project, the key performance indicators (KPI) are, generally, not measured.  However, it is at this stage of the project when you start thinking about what you should measure and how you will measure it(1).

First a word of warning, metrics are useful but they rarely provide the full picture(2).  That being said there are metrics that can be monitored

Bugs found at stage X…

One of the benefits of a Model-Based Design approach is the ability to detect bugs earlier on in the development process.bugs However, one side effect is the observation “we find so many bugs when following an MBD process.(3)”  Hence the metric that should be tracked is the number and severity of bugs found at each stage of the development process.  To determine the impact of finding the bugs early in the development a multiplier can be applied to the cost of the bug…

cost = Severity * StageConstant

Test and requirements coverage

Model-Based Design allows users to track the coveragePolyspacerequirements and testing coverage through automated tools(4).  With respect to test coverage; there are two types of test coverage.  The first is requirements based test coverage; do tests exist to cover the requirements.  The second are formal metrics coverage such as MCDC coverage.

The objective with coverage tracking is to see a steady increase in the percentage coverage over the development cycle.

Integration and development time

The final “primary” metrics are the development and integration time.  The development time is straight forward, how long from receipt of requirements to final check in of a model that satisfies the requirements (and test cases).  The integration time is a bit more interesting.

In an ideal workflow for Model-Based Design, there is integrationHandsan integration model that is created at the start of the development cycle.  Individual developers check their models against that integration model to ensure that there are no conflicts.  Hence in an ideal flow, the integration time should be near zero.

However, since there will be changes as the project develops, the integration model will change and the systems engineer will need to validate those changes.  Like the bug detection finding integration issues is done further upstream in an MBD process.  Again the metric should use a weighted value based on the stage of where the integration issue is found.

Final thoughts

This post covered what can be measured, not how to measure them; this will be covered in future posts.  Additional metrics can be covered however take care in having too many metrics and frustrating those who are developing with a heavy “tracking debit”.

Footnotes

(1) For this post, I am assuming that you do not currently have metrics in place.  If you have existing metrics they can be adapted to the MBD environment.
(2)Trying to capture all activities in development can, in fact, be detrimental in that it takes away time from development.  Always try to automate metric collection and, when not possible, simplify the process of creating this data.
(3) I have gotten this comment on many engagements; people mistake finding bugs in a process for not having bugs in their earlier process.  While there will be new bugs due to adopting a new process it is rare that an old process did not have any issues.
(4) Setting up the automation tools is something that is done in future steps of the adoption.

Model-Based Design foundational concepts

Published on by Michael BurkeLeave a comment

In previous blog posts, I have gone into some depth on testing, architecture and data management for models.  With this post, I will cover how these three activities for the foundation of any Model-Based Design process.

Foundations and core competencies

At later stages in the adoption of Model-Based Design crawl-walk-run-fly-300x191
processes task-specific groups will emerge (development, verification, and systems.)  However, at the start of the Model-BAsed Design process users from all groups need to determine the “common language” that will be used to develop their project (1).

Architecture and data sure, but why testing?

The identification of architecture and data as foundational concepts is generally well understood.modelcentricgifslow  Combined they define how people will develop the model through interfaces and clear communication.  So why testing in the trinity?  It returns to the core concept of Model-Based Design that models are at the center of the development process.  To ensure that the model can be used consistently through the development process they need to be “locked down”(2) with test cases.

Driven by this objective, the testing environment is designed at the start of the development process.  The requirements of the test environment should be addressed within the architectural and data infrastructure.  The good news is that best practices for the three “legs”(2) of the MBD stool are already in the well defined; it is a matter of honing them to your specific project and environment.

Final thoughts

This post is not intended to be technical; rather it is to remind us as we develop new processes to start out with the “best path forward” from the start.  In the section about the validation project, I will discuss the next round of tools that are commonly adopted.

Footnotes

(1) I may start using this graphic as my tag for MBD adoption.  Crawl = investigate.  Walk = initial.  Run = validated.  Flying = optimizing.
(2) Tests cases are elaborated as the model is developed.  The “lock down” is achieved through the use of a continuous build and test server.
(3) The metaphor of a tripod or stool can be overused.  But, to push it one last time, this is your stepping stool to the next round of MBD tools and processes.  Build it well and it forms a strong foundation.

Bug tracking software

Published on by Michael Burke5 Comments

The first thing I need to clarify in this post is, what is and what is not a bug.

A software bug is an error, flaw, failure or fault in a
computer program or system that causes it to produce
an incorrect or unexpected result, or to behave in
unintended ways.

A software bug is not is

  • Incomplete features:  During the software development process features will be under development(1).  As long as the incomplete nature of the feature does not introduce errors it is not considered a bug.
  • Desired features: Frequently the scope of a software develop project will not allow all the desired features to be included in a release of the software.  Again, as long as the lack of the feature does not introduce errors it is not considered a bug.

Incomplete features should already be tracked in the project planning timeline.  Desired features should be incorporated into the requirements document for the next generation of the project.

Severity of bugs

Not all bugs are created equal; defining the sevBeetle-Poster-720x479erity of bugs is necessary for prioritizing the correction of the bugs.  There are two common metrics for determining severity.  Frequency, how often does the bug occur.  Impact, when the bug occurs what happens to the program.  The following lists provide examples of how frequency and impact could be defined(2).

Frequency:

  • Infrequent:  Happens in less than 1% of the executions of the program in the normal work tasks.
  • Common: Happens for 1 ~ 5% of users in the normal course of work.
  • Prevalent: Occurs for 10% of users in their normal course of work.

Impact: 

  • Low:  Bugs that are cosmetic flaws or provide unclear information to the user. The user should be able to recover from these bugs without affecting their work.
  • Medium: Bugs that provide incorrect data to the user and or significantly impact the performance of the process.
  • High: Bugs that crash the program or create a loss of data for the user.

Once the frequency and impact have been determined the ranking of the bug can be defined.

Low Medium High
Infrequent Advised  Recommended Required
Common Advised Required Required
Prevalent Recommended Required Required

The ranks serve as a guideline for prioritizing bug fixes; with required, the recommended and finally advised bugs being fixed in that order(3).

Bug fixing workflow

There are multiple commercially available tools for bug tracking.  A basic workflow should include the following tasks and events.

bugFix

  • Bug detection:  The bug is found either through use or captured in an existing test case
  • Entry into tracking system: Once detected the bug, with comments and reproduction steps, should be entered into the bug tracking system(4).
  • Assignment:  The bug is assigned to a software engineer for resolution
  • Creation of test / validation of solution: If a test case does not already exist for the bug it should be created.  The proposed solution to the bug should be run against both the new test case and the existing test cases to ensure that the fix did not introduce new errors.

Final thoughts

Bug tracking and resolution is a problem common to all software development workflows.  The process for resolving these issues is the same for Model-Based Development as in traditional C development environments.  The critical part of bug resolution, as in all development, is that the bugs are clearly defined in an actionable fashion so that the test and software engineers can under stand the problem and find a solution.

Footnotes

(1)Feature encapsulation will help prevent incomplete features impacting other sections of the project.  See the software architecture posts for more information.
(2)These partial definitions for frequency and impact; depending on the type of system being developed the error types and frequencies should be adjusted.
(3)The table providing rankings based on frequency and impact should be adjusted depending on the type of system under development.  Additionally the criticality of some systems with in the whole should be taking into account when assigning impact.
(4)Entering bugs into the tracking system is critical for creating development metrics.  Without the entry there is no method for determining the efficiency of the overall process.