Difference between revisions of "Simulink Tester Issues"

From T-VEC Wiki
Jump to: navigation, search
Line 2: Line 2:
  
 
==Guidance==
 
==Guidance==
===Test Vector Failures===
+
===Test Vector Generation Failures===
 
If test vector failures occur
 
If test vector failures occur
 
#Check to make sure that the lower-level subsystems do not have failures, correct those failures first
 
#Check to make sure that the lower-level subsystems do not have failures, correct those failures first
 
#Check to make sure that the constraints of lower-level subsystem do not preclude satisfiability associated with higher-level subsystems that reference the lower-level subsystems
 
#Check to make sure that the constraints of lower-level subsystem do not preclude satisfiability associated with higher-level subsystems that reference the lower-level subsystems
 
#Use the Auto Rotate Convergence Fix Mode in the Vector Generator Properties of VGS
 
#Use the Auto Rotate Convergence Fix Mode in the Vector Generator Properties of VGS
 +
#Ensure the range settings of inputs variables do not preclude satisfiability
 
==Limitations==
 
==Limitations==
 
===Simulation Test Drivers: State Data Initialization===
 
===Simulation Test Drivers: State Data Initialization===
Line 39: Line 40:
 
The auto-rotate is one of the mechanisms that address this issue, most are applied automatically, independent of the specific convergence mode.  
 
The auto-rotate is one of the mechanisms that address this issue, most are applied automatically, independent of the specific convergence mode.  
 
Finding test vectors, by hand, for these cases is likely to be equally problematic.
 
Finding test vectors, by hand, for these cases is likely to be equally problematic.
 +
 +
===Signal Range Management===
 +
Changes in architectural decomposition may cause a signal name to be removed. For example, if a signal is defined as signal A in scope subsystems x\y\z and subsystem y is removed, then that signal is undefined and is removed from the signal range file (with extension .srf).
 +
 +
The translator creates a backup copy of the signal range file (.srf) each time the translator is run. It creates a log file that details every change to the .srf file. It should be possible to recover from an unexpected change to the .srf file.
 +
 +
===Model Coverage vs. Code Coverage===
 +
Test vector generation attempts to find test vectors for all of the paths (DCPs) in the model. Complete coverage means complete model coverage, however, complete model coverage does not guarantee coverage of code (e.g., MCDC coverage), for example: code generators can put in code (e.g., safety code around square root function). The Simulink Testers provides mechanisms such as assertions and coverage predicates that can be used to produce additional tests to cover many possible situations. Tools such as LDRA TBrun is integrated with the Simulink Tester and can provide test coverage measures to support organization that may need evidence to support FAA certification.
 +
 +
===Test Sequence Vectors===
 +
Many blocks in Simulink have specific initial condition values, unit delays, integrators, etc. (and stateflow in terms of things like initial states of state machines). T-VEC has a mechanism for using these initial values during vector generation. T=0 defines a point in time where all test vectors are generated as if the system is in the first cycle of execution. When option is T>=0 TSVs are generated for things that take place after the first cycle and then continue for multiple cycles of the TSV. When T>0 it turns off the initial condition setting action and allow the state variables to be solved for through convergence rather than be initilized/assigned.
 +
 +
[Image:State_Variable_Inits.jpg|center|VGS Property: State Variable Inits]
 +
 +
The default is to generate vectors for both cases, 2 vectors per DCP when no state variables are available, but T-VEC produces 4 vectors when state variables are available.

Revision as of 15:05, 25 February 2007

This section describes potential limitations, issues and associated guidance for using T-VEC with Simulink and Stateflow.

Contents

Guidance

Test Vector Generation Failures

If test vector failures occur

  1. Check to make sure that the lower-level subsystems do not have failures, correct those failures first
  2. Check to make sure that the constraints of lower-level subsystem do not preclude satisfiability associated with higher-level subsystems that reference the lower-level subsystems
  3. Use the Auto Rotate Convergence Fix Mode in the Vector Generator Properties of VGS
  4. Ensure the range settings of inputs variables do not preclude satisfiability

Limitations

Simulation Test Drivers: State Data Initialization

T-VEC produces test vectors that may contain state data for feedback constructs such as the unit delay. This results in state data, but there is currently no control mechanisms in the Matlab simulator to set (initialize) state data.

Issues

Inlining and Coverage

Use of inlining Simulink subsystems can impact test and model coverage. If a model includes a couple of references to division protection subsystems, which are inlined, then some, if not all, of the missing coverage is related to situations where the model doesn't support taking one of the paths through these utilities (lower-level subsystem) based simply on the model's design.

It is the very nature of design models vs requirements models. Designs include re-usable/generic components whose logic won't necessarily be needed for all applications. The only solution is to make sure they are treated as stand-alone atomic functions so that they may be tested in their own right.

Here's an example: Subsystem X is a utility that has an input A with two paths:

A >= 0 -> do something
A < 0 -> do something

Subsystem Y inlines subsystem X, but has references to Subsystem X only when the input associated with the signal A is >= 0. This means that there cannot be model or test coverage for the path when A < 0.

Test Vector Generation Failures

T-VEC VGS analysis is exhaustive in the sense that for every DCP throughout the hierarchy of systems it is model-checked, and a test vector is them produced. Most test vector generation failures results from model defects where the constraints of the DCP are not satisfiable throughout the hierarchy of DCPs of the subsystems.

However, there are possible situations where test vector generation fails when it is possible to find vectors, for example when signal domains are left to their defaults. There are default settings for ranges values, but these may not be adequate for the application.

T-VEC does cover a large percentage of every model use standard block, however, dynamic array indexing, and cases of spacial discontinuity coupled with circular dependent variable relationships that cause forward convergence to move into the "left hand" (output) disconnected space and backward propagation to move into "right hand" (input) disconnected space and no solution is possible with existing mechanisms. This is rather rare, but it does happen. The T-VEC test generation algorithms continue to evolve to deal with classes of these kind of situations, but there is no guarantee that there will never be a problem. For such situation contact support@t-vec.com for additional assistance.

The auto-rotate is one of the mechanisms that address this issue, most are applied automatically, independent of the specific convergence mode. Finding test vectors, by hand, for these cases is likely to be equally problematic.

Signal Range Management

Changes in architectural decomposition may cause a signal name to be removed. For example, if a signal is defined as signal A in scope subsystems x\y\z and subsystem y is removed, then that signal is undefined and is removed from the signal range file (with extension .srf).

The translator creates a backup copy of the signal range file (.srf) each time the translator is run. It creates a log file that details every change to the .srf file. It should be possible to recover from an unexpected change to the .srf file.

Model Coverage vs. Code Coverage

Test vector generation attempts to find test vectors for all of the paths (DCPs) in the model. Complete coverage means complete model coverage, however, complete model coverage does not guarantee coverage of code (e.g., MCDC coverage), for example: code generators can put in code (e.g., safety code around square root function). The Simulink Testers provides mechanisms such as assertions and coverage predicates that can be used to produce additional tests to cover many possible situations. Tools such as LDRA TBrun is integrated with the Simulink Tester and can provide test coverage measures to support organization that may need evidence to support FAA certification.

Test Sequence Vectors

Many blocks in Simulink have specific initial condition values, unit delays, integrators, etc. (and stateflow in terms of things like initial states of state machines). T-VEC has a mechanism for using these initial values during vector generation. T=0 defines a point in time where all test vectors are generated as if the system is in the first cycle of execution. When option is T>=0 TSVs are generated for things that take place after the first cycle and then continue for multiple cycles of the TSV. When T>0 it turns off the initial condition setting action and allow the state variables to be solved for through convergence rather than be initilized/assigned.

[Image:State_Variable_Inits.jpg|center|VGS Property: State Variable Inits]

The default is to generate vectors for both cases, 2 vectors per DCP when no state variables are available, but T-VEC produces 4 vectors when state variables are available.