Coverage Analysis

Purpose

The recording process has effectively generated an entire test script library covering over N% of the customers actual SAP system usage (where N would be expected to range from 70% to 90%). After the recording process is finished, the testing team can understand the level of coverage they have with the recording based upon the usage information of what the customer actually uses over the past M months (where M is typically 3 months). If key transactions are missing (e.g. month end processes) then the coverage analysis within Testimony can provide this information and additional scripts can be run to cover those processes that are missing.

There are a number of tabs in this functionality that perform the following functions:

1. Usage Retrieval

This is where the process begins and Usage Retrieval forms the basis of all other functions in the coverage analysis functionality of Testimony. Usage Retrieval is performed for all of the source systems in order to understand what the customer actually uses (and how often) over a configurable period. The default period for analysis is the last 3 months. A “usage retrieval run” is generated for a given source system in the current test plan. This usage retrieval is performed at any time prior to the customer running the playback process of Testimony. It forms the basis for the following two functions – auto object prioritization and the coverage analysis itself.

2. Prioritization

Prioritization is a method by which all objects in the system (e.g. RFC’s, dialog transactions). There are 3 aspects to prioritization which should be performed in the following order:

(1) Library based object prioritization
(2) Customer defined object prioritization
(3) Automatic object prioritization based up usage data

2.1 Library based object prioritization

Basis Technologies automatically delivers a set of object priorities that are out of the box. These are classified by type of system (ECC, CRM, BW etc) and then by application component. The majority of these are typically critical period end processes. These are core objects that run infrequently but are important from a regression testing environment.

2.2 Customer defined object prioritization

The determination of an objects priority is according to the following order:

- Automatic object prority
- Library defined object priority
- Customer defined object priority

This means that if a customer has defined the priority for a particular object (e.g. a month end report), then this will have the highest precedence over an object that is defined in the Testimony delivered library which then has priority of the automatically determined priority (based upon usage data).

Hence, in this pat of Testimony, the customer can define objects that have higher priority that what may have been automatically determined.

2.3 Automatic object prioritization

In order to determine the “criticality” of objects for the business, the core principal in this functionality is to look at how often it is used. The automatic object prioritization function in Testimony is run for a particular source system of the current test plan and for a previous “usage retrieval run” (as per the previous section 1). It is also possible to specify what percentage of frequently used objects are classified as “critical”, “high”, “medium” and “low”.

After the automated prioritization has been executed, all used objects will be automatically classified with a priority. This priority can be navigated and seen in the tree control in the lower half of the screen after selecting a particular source system. These priorities are broken down by the application component. It is possible to see both the automatically assigned priority plus whether this has been over-ridden by either the standard library or it has been customer defined.

3. Coverage Analysis

The final stage of the process is to execute the coverage run analysis itself. This is run for a particular source system, a selected usage retrieval run and based upon the contents of a selected execution queue. You can trigger this by selecting a particular system and then running the coverage analysis program.

The results of this run will provide you with a full coverage analysis of how closely the recorded transactions match what the usage data indicates the customer uses. It is possible to see the exact coverage percentage. For the missing objects (that are not covered) it is immediately visible what the missing % is by Critical, High, Medium and Low priority objects/processes.

Audience / Users

All Testimony users

Possible Actions

Run Usage Data Retrieval – Executes the usage data retrieval program which connects to the specified target system and retrieves the historical usage data for use in the coverage analysis.
Run Automated Object Prioritization – Automatically determines the prioritization of objects in the usage data based upon how often they are used.
Define Customer Object Prioritization – Allows the customer to over-ride the automatically determined object priorities for specific objects (usually for period end programs such as month end close).
Run Coverage Analysis – Executes the coverage analysis run to determine how much coverage is in the recorded transactions compared to what the customer uses in the live system(s).