Analyzing Coverage

Overview

The recording process has effectively generated an entire test script library covering over N% of the customer’s actual SAP system usage. The amount of coverage depends greatly on the length of the recording as well as the particular point in the business cycle when the recording was run. After the recording process is finished, the testing team can understand the level of coverage they have with the recording based upon the usage information of what the customer actually used over the past M months (where M is typically 3 months). If key transactions are missing (e.g. month end processes) then the coverage analysis within Testimony can provide this information. In this case, you can either test these processes outside of Testimony or look to include the periods when these processes run in your next Testimony recording.

There are a number of tabs in this functionality that perform the following functions:

1. Usage Retrieval

This is where the process begins and Usage Retrieval forms the basis of all other functions in the coverage analysis functionality of Testimony. Usage Retrieval is performed for all of the source systems in order to understand what the customer actually uses (and how often) over a configurable period. The SAP-standard period for retention of this information in the source system is three months. A “usage retrieval run” is generated for a given source system in the current test plan. This usage retrieval is performed at any time prior to the customer running the playback process of Testimony. It forms the basis for the following two functions – auto object prioritization and the coverage analysis itself.

2. Prioritization

Prioritization is a method by which all objects in the system (e.g. RFC’s, dialog transactions). There are 3 aspects to prioritization which should be performed in the following order:

(1) Library based object prioritization
(2) Customer defined object prioritization
(3) Automatic object prioritization based up usage data

2.1 Library based object prioritization

These are classified by type of system (ECC, CRM, BW etc) and then by application component. The majority of these are typically critical period end processes. These are core objects that run infrequently but are important from a regression testing environment.

2.2 Customer defined object prioritization

The determination of an object’s priority is according to the following order:

- Automatic object prority
- Library defined object priority
- Customer defined object priority

This means that if a customer has defined the priority for a particular object (e.g. a month end report), then this will have the highest precedence over an object that is defined in the Testimony delivered library which then has priority over the automatically determined priority (based upon usage data).

Hence, in this part of Testimony, the customer can define objects that have higher priority than what may have been automatically determined.

2.3 Automatic object prioritization

In order to determine the “criticality” of objects for the business, the core principal in this functionality is to look at how often it is used. The automatic object prioritization function in Testimony is run for a particular source system of the current test plan and for a previous “usage retrieval run” (as per the previous section 1). It is also possible to specify what percentage of frequently used objects are classified as “critical”, “high”, “medium” and “low”.

After the automated prioritization has been executed, all used objects will be automatically classified with a priority. This priority can be navigated and seen in the tree control in the lower half of the screen after selecting a particular source system. These priorities are broken down by the application component. It is possible to see both the automatically assigned priority plus whether this has been over-ridden by either the standard library or it has been customer defined.

3. Coverage Analysis

The final stage of the process is to execute the coverage run analysis itself. This is run for a particular source system, a selected usage retrieval run and based upon the contents of a selected execution queue. You can trigger this by selecting a particular system and then running the coverage analysis program.

The results of this run will provide you with a full coverage analysis of how closely the recorded transactions match what the usage data indicates the customer uses. It is possible to see the exact coverage percentage. For the missing objects (that are not covered) it is immediately visible what the missing % is by Critical, High, Medium and Low priority objects/processes.