Clinical Research Oriented Workshop (CROW) Meeting: June 4, 2020

Present:   Levi Bonnell, Jessica Clifton, Justine Dee, Steve DeVoe, Nancy Gell, Juvena Hitt, Amanda Kennedy, Ben Littenberg, Jen Oshita, Bradley Tompkins, Connie van Eeghen, Adam Sprouse-Blum, Mariana Wingood

1.                   Warm Up: Justine got scooped ☹

2.                   Ben: Chalk talk on sensitivity and specificity, ROC curves, and more

a.       Deck of cards, with 37% “sick patients” denoted with red backs

                                                   i.      Picture card means positive test

                                                 ii.      Non-picture card means negative test

b.       Threshold: minimum value to call the test positive, with aces reporting negative and kings reporting positive

c.       Prevalence: those with the disease/N

d.       Those who have the disease:

                                                   i.      True positive + false negative

e.       Sensitivity: ability to correctly identify the sick (true positive and false negative)

                                                   i.      True Positive Rate: true positive/(true positive + false negative)

f.        Specificity: ability to correctly identify the well (false positive and true negative)

                                                   i.      True Negative Rate: true negative/(true negative + false positive)

g.       Positive Predictive Value: correctness of a positive test: D+|T+ = TP/(TP+FP)

h.       Negative Predictive Value: correctness of a negative test: D-|T- = TN/(TN+FN)

i.         Example with card deck

                                                   i.      Of the 19 reds (who truly have disease)

1.       11 have a correct positive test

2.       8 have a false positive test

                                                 ii.      Of the 33 blues (who truly do not have disease)

1.       1 has a false positive test

2.       32 have a correct positive test

                                               iii.      Positivity: 12/52 à how many positives will come from a testing process that will need further services.  Also, “test positivity” IS NOT the accuracy of the test or the prevalence

                                               iv.      True Positive Rate: 11/19 = 57.9% - not very good at identify the ill

                                                 v.      True Negative Rate: 32/33 = 97% - very good at identifying the well

                                               vi.      Positive Predictive Value: 11/12 = 91.7%

                                              vii.      Negative Predictive Value: 32/40 = 80%

j.         The placement of the threshold drives the above values

                                                   i.      Test manufacturers can set these with known data, to create a numeric threshold; some thresholds are based on analogue variables or based on subjective assessments

k.       Receiver Operating Characteristics Curve

                                                   i.      False Positive Rate on x axis

                                                 ii.      True Positive Rate on the y axis

1.       Goal is to have a high TPR and low FPR

                                               iii.      AUC: area under the ROC

1.       A diagonal ROC (straight line diagonal) means the test is 50% likely to get a true positive and 50% likely to get a true negative

2.       A ROC below the represents a bad test; a perfect test is represented by the axes

3.       For covid, we want high sensitivity at the cost of specificity – get the public off the streets

a.       For a painful, costly treatment, we want high specificity – don’t hurt people needlessly

b.       Outside the ROC are the people who get missed by the test

l.         PowerPoint and Excel sheet coming to an inbox near you soon!

3.                   Future sessions

a.       Next week: Justine’s topic and the all claims data base

b.       Future topics: Jessica’s article in the Cone of Silence