Difference between revisions of "Simulink Tester Issues"

From T-VEC Wiki
Jump to: navigation, search
(Math Functions)
 
Line 3: Line 3:
 
==Guidance==
 
==Guidance==
 
===Test Vector Generation Failures===
 
===Test Vector Generation Failures===
If test vector failures occur
+
If test vector failures occur there are some general guidelines that apply to any type of model (TTM or Simulink). See [[VGS Issues and Guidelines|VGS Issues and Guidelines]].
#Check to make sure that the lower-level subsystems do not have failures, correct those failures first
+
 
#Check to make sure that the constraints of lower-level subsystem do not preclude satisfiability associated with higher-level subsystems that reference the lower-level subsystems
+
===S-Functions===
#Use the Auto Rotate Convergence Fix Mode in the Vector Generator Properties of VGS
+
S-Functions allow users to specify the semantics of new blocks for simulation and code generation through several programming languages, including MATLAB, C, C++, Ada, and Fortran. While the blocks’ semantics are necessary to support test generation for subsystems that reference the S-Function, these semantics are not accessible by the translator through the standard approach.
#Ensure the range settings of inputs variables do not preclude satisfiability
+
 
 +
To support translation of S-Functions, the translator allows for the substitution of a Simulink subsystem in place of a S-Function. The subsystem that replaces the S-Function describes the functionality of the S-Function in terms of primitive Simulink blocks. The Simulink subsystem to be substituted for the S-Function is specified by entering a specially formated tag into the comment of the S-Function block.
 +
 
 +
Instructions for formating the S-Function substitution tag are provided in the Simulink Tester User's Guide.
 +
 
 +
The S Function must be created to use S Function Builder that can be supported by T-VEC.
 +
 
 +
S-functions are used by Simulink developers to write source code and include it in their Simulink models. We do not directly support using source code as an input to the T-VEC translator. That would defeat the main purpose of the sl2tvec/T-VEC process - which is to support model-based verification and testing. Source code is not a "model" and is the target that the user is supposed to be testing. It is not valid to generate test vectors directly from the source code being tested.
 +
 
 +
The way that we support S-functions is to allow the user to create an independent Simulink model that represents the input and output interface to the S-function and also represents the same functional semantics as the S-function. Then, the user can add a reference to the S-function in the block properties of the S-function block that names the external Simulink model. The sl2tvec translator then substitutes the external Simulink model for the S-function during translation. This way, the user can use S-functions for code efficiency or direct addressing of specific memory locations or for what ever reason they are necessary, but still use the sl2tvec/T-VEC processes on their model.
 +
 
 +
----
 +
 
 +
===Embedded Matlab Language (EML) Translation: m-script guidelines===
 +
Support for the translation of Embedded Matlab Language (EML) functions is now available. Not all EML functions will be immediately available but they will be scheduled for implementation as requested. Only those EML functions that can be graphically represented using Simulink blocks can be implemented at this time. Support for other EML functions can be added but they may require a development contract. If there is a function you need please E-mail your request to support@t-vec.com.
 +
====M-Scripting Guidance====
 +
A list of the currently supported EML functions is included in this wiki. Additionally there are a few programming shortcuts that are not supported. These are listed in the Unsupported Syntax section. If your script uses those shortcuts then minor modification will be required.
 +
 
 +
=====Global Variables=====
 +
Global variables should not be used in your EML script. All external values needed by a function should be passed into that function as parameters.
 +
 
 +
=====Workspace variables=====
 +
Because the processing of the EML script is done outside of Matlab workspace variables should be passed in as function parameters if needed.
 +
 
 +
====Unsupported syntax====
 +
=====Matrix field selection using a single parameter=====
 +
Matrix indexing with 1 parameter is not supported.
 +
<br>
 +
m=[1,2,3;4,5,6];
 +
<br>
 +
x=m(2); % not supported
 +
<br>
 +
<br>
 +
Instead, use row and column syntax for matrix indexing.
 +
<br>
 +
m=[1,2,3;4,5,6];
 +
<br>
 +
x=m(2,1);
 +
<br>
 +
=====Defining a variable and using it outside of the block containing it=====
 +
The following syntax is not supported.
 +
<br>
 +
function a=fcn1(x)
 +
<br>
 +
if (x==4)
 +
<br>
 +
&nbsp;&nbsp;s=3;  % variable s is initialized inside the if block
 +
<br>
 +
end;
 +
<br>
 +
a=x+s; %translation error: variable s is used before it has been initialized
 +
<br>
 +
<br>
 +
Instead, declare variables in the scope they will be used.
 +
<br>
 +
function a=fcn1(x)
 +
<br>
 +
s=0;  % variable s is initialized at the function scope
 +
<br>
 +
if (x==4)
 +
<br>
 +
&nbsp;&nbsp;s=3;  % ok
 +
<br>
 +
end;
 +
<br>
 +
a=x+s; %ok
 +
<br>
 +
<br>
 +
Using the unsupported syntax will cause the following error to be generated:
 +
<br>
 +
ERROR SL0251: (Internal) Unknown variable found while processing embedded Matlab script.
 +
<br>
 +
&nbsp;&nbsp;This is usually caused when a variable is used outside of the scope it was declared within.
 +
<br>
 +
&nbsp;&nbsp;Variable will be grounded. 
 +
<br>
 +
&nbsp;&nbsp;identifier : [identifier]
 +
<br>
 +
&nbsp;&nbsp;scope : [block location]
 +
<br>
 +
&nbsp;&nbsp;line: [line] pos: [column]
 +
<br>
 +
 
 +
====Supported EML syntax====
 +
We will be adding more function support with future releases. If you call an EML function that has not been implemented then you will see a message like:
 +
<br>
 +
<br>
 +
ERROR SL0253: (Internal) No subsystem implementation found. 
 +
<br>
 +
No subsystem implementation found for subsystem reference: [function name].
 +
<br>
 +
&nbsp;&nbsp;Please verify that your model passes the Matlab model check (CTRL+D). 
 +
<br>
 +
&nbsp;&nbsp;If your model is correct then you could be calling a function that has not been implemented in the T-VEC Simulink tester.
 +
<br>
 +
&nbsp;&nbsp;If you believe this to be the case, please send an E-mail to support@t-vec.com with the function name so that we can include it in a future release.
 +
<br>
 +
=====Flow control syntax=====
 +
if, elseif, else
 +
=====Conditional syntax=====
 +
&&,||  %and, or
 +
 
 +
=====Relational syntax=====
 +
<, <=,==,~=,>,>=  %less than, less than or equal, equal, not equal, greater than, greater than or equal
 +
=====Math syntax=====
 +
+,- %addition, subtraction
 +
<br>
 +
^  %power of
 +
<br>
 +
<nowiki>*</nowiki>,/  %mulitplication, division
 +
=====Unary syntax=====
 +
+,-
 +
=====Matrix syntax=====
 +
[r1c1,r1c2,r1c3;r2c1,r2c2,r2c3;]
 +
<br>
 +
[matrix]' %transpose matrix
 +
=====Comment syntax=====
 +
% [...] - single line comment
 +
<br>
 +
%{
 +
<br>
 +
[...] - multiline comment block
 +
<br>
 +
%}
 +
=====Line continuation syntax=====
 +
...  %line continuation
 +
 
 +
====Supported EML functions====
 +
Below is a list of the currently supported EML functions.
 +
=====Rounding Functions=====
 +
floor
 +
ceil
 +
round
 +
fix
 +
 
 +
=====Trigometric Functions=====
 +
sin
 +
cos
 +
tan
 +
asin
 +
acos
 +
atan
 +
atan2
 +
 
 +
=====Math Functions=====
 +
abs
 +
dot
 +
exp
 +
hypot
 +
log
 +
log10
 +
mod
 +
power
 +
rem
 +
sqrt
 +
 
 +
=====Matrix Functions=====
 +
eye
 +
inv
 +
 
 +
----
 +
 
 
==Limitations==
 
==Limitations==
 
===Simulation Test Drivers: State Data Initialization===
 
===Simulation Test Drivers: State Data Initialization===
T-VEC produces test vectors that may contain state data for feedback constructs such as the unit delay. This results in state data, but there is currently no control mechanisms in the Matlab simulator to set (initialize) state data.  
+
T-VEC produces test vectors that may contain state data for feedback constructs such as the unit delay. This results in state data, but there is currently no control mechanisms in the Matlab simulator to set (initialize) state data.
 +
 
 
==Issues==
 
==Issues==
 
===Inlining and Coverage===
 
===Inlining and Coverage===
 
Use of inlining Simulink subsystems can impact test and model coverage.  
 
Use of inlining Simulink subsystems can impact test and model coverage.  
If a model includes a couple of references to division protection
+
If a model includes inlined subsystems, then the missing coverage  
subsystems, which are inlined, then some, if not all, of the missing coverage  
+
may be related to situations where the model doesn't support taking one of  
is related to situations where the model doesn't support taking one of  
+
the paths through the utilities (lower-level subsystem) based simply on the model's design.  
the paths through these utilities (lower-level subsystem) based simply on the model's design.  
+
  
 
It is the very nature of [[Simulink Tester for T-VEC#Design Model|design models]] vs [[T-VEC_Tablular_Modeler#Requirement Modeling|requirements models]]. Designs include  
 
It is the very nature of [[Simulink Tester for T-VEC#Design Model|design models]] vs [[T-VEC_Tablular_Modeler#Requirement Modeling|requirements models]]. Designs include  
 
re-usable/generic components whose logic won't necessarily be needed for  
 
re-usable/generic components whose logic won't necessarily be needed for  
all applications. The only solution is to make sure they are treated as
+
all applications.  
stand-alone atomic functions so that they may be tested in their own right.
+
  
 
Here's an example:
 
Here's an example:
 
Subsystem X is a utility that has an input A with two paths:
 
Subsystem X is a utility that has an input A with two paths:
  
: A >= 0 -> do something
+
: A >= 0 then do something
: A < 0  -> do something
+
: A < 0  then do something
  
 
Subsystem Y inlines subsystem X, but has references to Subsystem X only when the input associated with the signal A is >= 0. This means that there cannot be model or test coverage for the path when A < 0.
 
Subsystem Y inlines subsystem X, but has references to Subsystem X only when the input associated with the signal A is >= 0. This means that there cannot be model or test coverage for the path when A < 0.
 +
 +
The only solution is to make sure they are treated as stand-alone atomic functions so that they may be tested in their own right.
 +
 
===Test Vector Generation Failures===
 
===Test Vector Generation Failures===
T-VEC VGS analysis is exhaustive in the sense that for every DCP throughout the hierarchy of systems it is model-checked, and a test vector is them produced. Most test vector generation failures results from model defects where the constraints of the DCP are not satisfiable throughout the hierarchy of DCPs of the subsystems.
+
T-VEC VGS analysis is exhaustive in the sense that for every [[Modeling#Low Level Representation|DCP]] throughout the hierarchy of systems is model-checked, and a test vector is then produced. Most test vector generation failures result from model defects where the constraints of the DCP are not satisfiable throughout the hierarchy of DCPs of the subsystems.
  
However, there are possible situations where test vector generation fails when it is possible to find vectors, for example when signal domains are left to their defaults. There are [http://www.t-vec.com/wiki/index.php/Simulink/T-VEC_Examples default settings for ranges values], but these may not be adequate for the application.
+
However, there are possible situations where test vector generation fails when it is possible to find vectors. For example, when signal domains are left to their defaults. There are [[Simulink/T-VEC_Examples|default settings for ranges values]], but these may not be adequate for the application.
  
T-VEC does cover a large percentage of every model use standard block, however, dynamic array indexing, and cases of spacial discontinuity coupled with circular dependent variable relationships that cause forward convergence to move into the "left hand" (output) disconnected space and backward propagation to move into "right hand" (input) disconnected space and no solution is possible with existing mechanisms. This is rather rare, but it does happen. The T-VEC test generation algorithms continue to evolve to deal with classes of these kind of situations, but there is no guarantee that there will never be a problem. For such situation contact support@t-vec.com for additional assistance.
+
T-VEC does cover a large percentage of models that use standard blocks (see View | Library... from Simulink Tester GUI for the supported blocks), however, there are other situations where T-VEC may not produce vectors, for example
 +
*dynamic array indexing
 +
*spacial discontinuity coupled with circular dependent variable relationships
  
The auto-rotate is one of the mechanisms that address this issue, most are applied automatically, independent of the specific convergence mode.  
+
The Auto Rotate Convergence Fix Mode Test Vector Generation property is one of the mechanisms that can address some of these issues, and other internal mechanisms are applied automatically, independent of the specific convergence mode. The T-VEC test generation algorithms continue to evolve to deal with classes of these kind of situations, but there is no guarantee that such situations will not occur. For such situation contact support@t-vec.com for additional assistance.
Finding test vectors, by hand, for these cases is likely to be equally problematic.
+
  
 
===Signal Range Management===
 
===Signal Range Management===
Changes in architectural decomposition may cause a signal name to be removed. For example, if a signal is defined as signal A in scope subsystems x\y\z and subsystem y is removed, then that signal is undefined and is removed from the signal range file (with extension .srf).
+
Changes in architectural decomposition may cause a signal name to be removed. For example, if a signal is defined as signal A in the hierarchical scope of subsystems x\y\z and subsystem y is removed, then that signal is undefined and is removed from the signal range file (with extension .srf).
  
 
The translator creates a backup copy of the signal range file (.srf) each time the translator is run. It creates a log file that details every change to the .srf file. It should be possible to recover from an unexpected change to the .srf file.
 
The translator creates a backup copy of the signal range file (.srf) each time the translator is run. It creates a log file that details every change to the .srf file. It should be possible to recover from an unexpected change to the .srf file.
  
 
===Model Coverage vs. Code Coverage===
 
===Model Coverage vs. Code Coverage===
Test vector generation attempts to find test vectors for all of the paths (DCPs) in the model. Complete coverage means complete model coverage, however, complete model coverage does not guarantee coverage of code (e.g., MCDC coverage), for example: code generators can put in code (e.g., safety code around square root function). The Simulink Testers provides mechanisms such as assertions and coverage predicates that can be used to produce additional tests to cover many possible situations. Tools such as LDRA TBrun is integrated with the Simulink Tester and can provide test coverage measures to support organization that may need evidence to support FAA certification.
+
Test vector generation attempts to find test vectors for all of the paths (DCPs) in the model. Complete coverage means complete [[Model_Coverage|model coverage]], however, complete model coverage does not guarantee coverage of code (e.g., MC/DC coverage), for example: code generators can put in code for checking and handling exceptional conditions (e.g., safety code around square root function). The Simulink Testers provides mechanisms such as assertions and coverage predicates that can be used to produce additional tests to cover many possible situations. Tools such as LDRA TBrun is integrated with the Simulink Tester and can provide test [[Model_Coverage#Code Coverage|coverage measures]] to support organizations that may need evidence to support FAA certification.
 +
 
 +
===Test Sequence Vectors: State Variable Initialization===
 +
Test Sequence Vectors (TSVs) are created by referencing the target subsystem multiple times.
 +
Special T-VEC subsystems are created based on a configuration file supplied to the translator. These TSV subsystems include one or more test sequences. Each test sequence
 +
includes one or more references to the target subsystem (i.e., the number of sample times
 +
being tested). It includes also a set of the target subsystem’s inputs for each time that the
 +
target subsystem is referenced. These input values have their respective subsystem reference
 +
appended to their name. Therefore, if a subsystem has two inputs, inputA and inputB,
 +
then a TSV subsystem that includes two references would include the inputs inputA_1,
 +
inputB_1, inputA_2 and inputB_2. Input values to each subsystem reference can be
 +
controlled through settings at the reference level, test sequence level or subsystem level.
 +
 
 +
Many blocks in Simulink have specific initial condition values, unit delays, integrators, etc. (and stateflow in terms of things like initial states of state machines). T-VEC has a mechanism for using these initial values during vector generation, and they include:
 +
*T=0 - defines a point in time where all test vectors are generated as if the system is in the first cycle of execution
 +
*T>=0 - test vectors are generated for things that take place after the first cycle and then continue for multiple cycles of the TSV
 +
*T>0 - test vectors where the initial condition setting action is disabled and this forces the state variables to be solved for through convergence rather than be initilized or assigned
 +
*Ignore
 +
 
 +
[[Image:State_Variable_Inits.jpg|center|VGS Property: State Variable Inits]]
 +
 
 +
The default is to generate vectors for both cases, 2 vectors per DCP when no state variables are available, and 4 test vectors when state variables are available. This mechanism should be adequate to test control and logic associated with state data (See [[State Variable Inits|State Variable Inits Example]]).
 +
 
 +
===Test Driver Generation===
 +
There are some situations where the object mapping information that relates a signal name to an implementation name must be defined manually to support test driver generation. This is applicable to a few types of Simulink blocks and some Stateflow modeling constructs. There are cases where it is simply not possible to predict variable names, function parameter orders, etc. There is no API for determining the choices made by the RTW to fully automate all aspects of test driver map file creation.
 +
 
 +
There is a work around with the current schemas - all mapping descriptors are passed through a Perl function before going into the target file. The perl script parses in an external file that contains new replacement mapping information. The mapping can be defined with the descriptor or a mapping key. If the perl script finds a match then it uses the replacement from the file.  This function can be extended directly through the Perl script.
  
===Test Sequence Vectors===
+
===Project Differences when Translating with Mappings===
Many blocks in Simulink have specific initial condition values, unit delays, integrators, etc. (and stateflow in terms of things like initial states of state machines). T-VEC has a mechanism for using these initial values during vector generation. T=0 defines a point in time where all test vectors are generated as if the system is in the first cycle of execution. When option is T>=0 TSVs are generated for things that take place after the first cycle and then continue for multiple cycles of the TSV. When T>0 it turns off the initial condition setting action and allow the state variables to be solved for through convergence rather than be initilized/assigned.
+
By default the function name option for a subsystem is set to “Auto”. The routine RTW uses to derive these function names is unknown to sl2tvec.  As a result, when translating from just the model file, the translator must guess what the function names are, but when translating with the model and RTW mappings file, the translator can determine from the mappings file what the function names are. In the case of reusable subsystems, the reusable subsystem definitions are only combined if they resolve to the same function name.  Since the sl2tvec translator guesses at these function names, the reusable subsystems may be combined differently if the mappings file in included in translation. The result is that a project translated with mappings may have a different number of reusable subsystems (or different subsystem names) than a project translated without mappings.
 +
 +
===Impact of Out of Date Signals===
 +
The translator uses the signal labels in a model to perform the translation process. Sometimes these labels can get out of date when signals are set to show their propagated names. This happens mainly a signal label is changed from one name to another, or when a library is moved from one location to another where the propagated signal label no longer applies.  
 +
When the labels get out of date, the translator cannot properly translate the model and produces error messages such as this:
  
[Image:State_Variable_Inits.jpg|center|VGS Property: State Variable Inits]
+
: Determining signal dimensions..ERROR SL0086: 
 +
: Signal dimensions are not consistent for block "Bus_Selector" in subsystem "example" (block expects an input labeled signal2, but could not find it in the input signal)
 +
: WidthIndex 1, BusLabel signal1, Output signal2
 +
: WidthIndex 2, BusLabel signal3, Output signal2
 +
: ERROR SL0115: signal dimension is invalid for signal "signal2" from block "Bus_Selector" <dimension: 0>
 +
: ..Done.
 +
 +
: Translation Aborted!
  
The default is to generate vectors for both cases, 2 vectors per DCP when no state variables are available, but T-VEC produces 4 vectors when state variables are available.
+
To avoid this error message, make sure a model is up to date before exporting it by selecting the Edit | Update Diagram menu item or pressing Ctrl + D.

Latest revision as of 20:53, 21 July 2009