One of the challenges with Testimony producing a clear set of defects that are purely connected to the release being tested is the “noise” that can exist in target systems from slight differences in environmental setup of these systems. While suppressions and exclusions can help, this can take time to setup to have the desired impact and runs the risk of masking genuine regression defects.
The Double Playback option helps to filter out this “noise”. This operates by running a “baseline” playback into an identical copy of the source system, with no changes or release candidate applied. The first execution queue is then copied and a second playback is then required. This time the identical copy of the source system is upgraded or has the changes or release candidates applied. The failures in the second (release) queue are compared to the baseline. If the failures occurred in the second playback for the same reason, these are excluded from defect proposal generation for the second queue therefore reducing the noise. The overview of the steps to perform a Double Playback are show in the diagram below:
The comparison of the two playback runs occurs with the Defect Proposal
Select a different variant prior to the Defect Proposal.
Ensure that the Double Playback check box is selected and enter the Execution Queue ID of the “Baseline” against the Double Playback Baseline Queue
Post your comment on this topic.