Subscribe to RSS
DOI: 10.3766/jaaa.24.10.2
Understanding Protocol Performance: Impact of Criterion and Test Correlation
Publication History
Publication Date:
06 August 2020 (online)
Background: A test protocol is created when individual tests are combined. Protocol performance can be calculated prior to clinical use; however, the necessary information is seldom available. Thus, protocols are frequently used with limited information as to performance. The next best strategy is to base protocol design on available information combined with a thorough understanding of the factors that determine protocol performance. Unfortunately, there is limited information as to these factors and how they interact.
Purpose: The objective of this article and the next article in this issue is to examine in detail the three factors that determine protocol performance: (1) protocol criterion, (2) test correlation, (3) test performance. This article examines protocol criterion and test correlation. The next article examines the impact of individual test performance and summarizes the results of this series. The ultimate goal is to provide guidance on the formulation of a protocol using available information and an understanding of the impact of these three factors on performance.
Research Design: A mathematical model is used to calculate protocol performance for different protocol criteria and test correlations while assuming that all individual tests in the protocol have the same performance. The advantages and disadvantages of the different criteria are evaluated for different test correlations.
Results: A loose criterion will produce the highest protocol hit and false alarm rates; however, the false alarm rate may be unacceptably high. A strict criterion will produce the smallest protocol hit and false alarm rates; however, the hit rate may be unacceptably low. Adding tests to a protocol increases the probability that the protocol false alarm rate will be too high with a loose criterion and that the protocol hit rate will be too low with a strict criterion. The intermediate criterion, about which little has been known, provides advantages not available with the other two criteria. This criterion is much more likely to produce acceptable protocol hit and false alarm rates. It also has the potential to simultaneously produce a protocol hit rate higher, and a false alarm rate lower, than the individual tests. The intermediate criteria produce better protocol performance than the loose and strict criteria for protocols with the same number of tests. For all criteria, best protocol performance is obtained when the tests are uncorrelated and decreases as test correlation increases. When there is some test correlation, adding tests to the protocol can decrease protocol performance for a loose or strict criterion. The ability of a protocol to manipulate hit and false alarm rates, or improve performance relative to that of the individual tests, is reduced with increasing test correlation.
Conclusions: The three criteria, loose, strict, and intermediate, have definite advantages and disadvantages over a large range of test correlations. Some of the advantages and disadvantages of the loose and strict criteria are impacted by test correlation. The advantages of the intermediate criteria are relatively independent of test correlation. When three or more tests are used in a protocol, consideration should be given to using an intermediate criterion, particularly if there is some test correlation. Greater test correlation diminishes the advantages of adding tests to a protocol, particularly with a loose or strict criterion. At higher test correlations, fewer tests in the protocol may be appropriate.