Clinical Research Oriented Workshop (CROW) Meeting: March 22, 2012

Present: Kairn Kelley, Ben Littenberg, Charlie MacLean, Connie van Eeghen

1. Check In: Cleveland may rock, but it’s a long way from Boston and it’s NOT Boston.

2. Presentation: Kairn – Next step is an article analyzing inter-rater scoring reliability; need feedback on structure and process.
a. This article will support the future study planned around the goodness of the tests being compared.
b. Research question: What is the inter-rater reliability (IRR) for scoring the dichotic words test in a convenience sample of children aged 6 – 11 years of age. Two certified clinicians, who were present for the same test, scored whether the child stated both words and if incorrect, what the word the child spoke was. Each child was tested with 100 pairs of words from a test not publically available and still under development. The test scores were entered in Excel and run through Stata. Data about children:
i. Age (6-11)
ii. Some sibling pairs
iii. Left/right handed
iv. Hearing sensitivity (mostly normal)
v. No known ADD
c. Definition of IRR:
i. The number of target words that the raters agreed on (binary: yes/no): proportion. Hard to interpret, unless there is a pattern. (n=8000) It evaluates the raters, not the kids.
ii. Number of target words agreed on: proportion (n=200). Did the raters rate the words the same.
iii. Mean score agreement by word (comparing average scores for each word) (Kairn had a great graph that generated this) – shows variability within the word list and could answer the question “what are the best words” – which is NOT the research question: mean difference. (n=200) Evaluates the words more than the raters. Did the raters agree on average.
iv. Mean score by subject: mean difference (n=40). At what point is the difference is important? Coefficient of variation (CV): the difference/standard deviation. However, the population of scores does not have a normal distribution, but there is no non-parametric equivalent to the CV.
v. By pass/file – to identify the threshold for passing the test proportion (n=40). This is the test for diagnosis – it affects clinical behavior. However, an accepted threshold does not yet exist and the children were not necessarily homogenous in hearing ability.
d. Tables that will answer the question: definitions 2 and 3 and 4 (as a summary) above.
i. For each word, the two raters P/F combinations A:P/F and K:P/F
ii. % P of A
iii. %P of K
iv. %P of mean (where P represents Pass)
v. # agree
vi. # disagree
vii. Mean difference
viii. % Agree
ix. Kappa
x. CI%
xi. P value
e. The tables themselves can start with the total number of words (200) for each of the above (plus the table total “All”), and then focus on the best and worst IRR scores.
f. The next layer is the total number of subjects/children (40). Subgroups:
i. Gender (M/F)
ii. Age (Old/Young by median) – or grade in school
iii. Handedness (L/R)
iv. High/Low scores (by median)
v. Hearing loss/no hearing loss
g. Other tables
i. Description of subjects
ii. Subject flow (how many invited, how many accepted, how many eligible, how many completed, …)
iii. “What if” tables – impact of dropping the hard words (which would improve the standard deviation) – is this a separate paper?
iv. IRR of recorded utterances – also another paper.
h. Draft article to Seminar on Friday, May 3

3. Workshop Goals for 2012:
a. Journal club: identify UVM guests and articles; invite to CROW ahead of time
b. Research updates: share work-in-process

4. Next Workshop Meeting(s): Thursday, 1:00 p.m. – 2:30 p.m., at Given Courtyard Level 4.
a. Mar 29: Charlie: Writers’ workshop Errors in Screening for Diabetic Kidney Disease
b. Apr 5: Rodger (topic TBD, perhaps Wilson Pace as guest?) or Steven Schroeder, MD: Improving Health in America – Why Change Comes So Hard from 12:00 – 1:30 in Davis Auditorium; reception to follow in HSRF 100, Hoehl Gallery ???
c. Apr 12: ask Amanda for a Journal Article (no Connie)
d. Apr 19: Kairn: draft article on Inter-Rater Reliability
e. Apr 26:
f. Future agenda to consider:
i. Ben: budgeting exercise for grant applications
ii. Ben: Writer’s workshop on the effect of the built environment on BMI (Littenberg & Austin Troy)
iii. Journal Club: “Methods and metrics challenges of delivery-system research,” Alexander and Hearld, March 2012 (for later in the year)
iv. Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding. Also: discuss design for PCBH clinical and cost research.
v. Amanda: presentation and interpretation of data in articles
vi. Sharon Henry: article by Cleland, Thoracic Spine Manipulation, Physical Therapy 2007
vii. Future: Review of different types of journal articles (lit review, case study, original article, letter to editor…), when each is appropriate, tips on planning/writing (Abby)

Recorder: C. van Eeghen