Clinical Research Oriented Workshop (CROW) Meeting: Nov 1, 2012

Present: Kairn Kelley, Ben Littenberg, Charlie MacLean, Prema Menon, Connie van Eeghen

1.                  Start Up: Post Sandy-Hurricane – at least it’s not raining.

2.                  Presentation: Kairn: final draft of article on IRR titled: “Inter-rater reliability for scoring dichotic words test responses of 6-10 year-old children.”  The group reviewed the draft and made the following comments/suggestions:

a.       Target journal: American Speech-Language-Hearing Association journal (American Journal of Audiology) or Journal of the American Academy of Audiology. Content of journal is consistent with this draft. Add more explanation re: why a logistic regression was used (Bonferroni correction details irrelevant after change to analysis discussed later; Bonferroni adjusts for likelihood of statistical significance occurring by chance from multiple comparisons)

b.      State sooner that the test’s word list is novel, although the test concept is well-established.  Consider saying more about why this new word list was developed.

c.       Abstract states IRR improves over time, but this doesn’t explain the duration involved.  These were naïve raters for this word list; worth saying up front.

d.      A detailed discussion on the two alternative figures ensured, which was so engaging that the recorder forgot to take notes.  Decision: any words outside the 95% confidence interval should be considered for exclusion from the word list.

e.       Consider doing a clustered analysis, as there is randomness among words and randomness among children. 

f.       Logistical regression results: keep the details that show change by session, which was the significant variable in explaining change in rater performance.  Include age, sex, and hearing in the data table.  Experiment with a figure, showing the change in session over time (possibly a series of box plots).

g.      Methods decision to use percent agreement might be the place to explain why Kappa was not used.  Or a sentence, in discussion, as a strength, rather than in the Limitations section.

h.      Recheck the language for the section marked: Test Item Difficulty/Pass Proportion

i.        Ben has the best tongue twisters in the group, but Charlie came up with some good ones too.

j.        Next: redraft – and think about how to do this for ADHD children.

3.                  Next Workshop Meeting(s): Thursday, 2:00 p.m. – 3:30 p.m., at Given Courtyard Level 4. 

a.       Nov 8: Rodger & Connie: presentation on Strategies on Implementing BHI

b.      Nov 15: Abby: review of draft manuscript for NAS predictors article (no Ben)

c.       Nov 22: THANKSGIVING… take the day off J

d.      Nov 29: Prema: draft grant application

e.       Dec 6:

f.       Future agenda to consider:

                                                  i.      Kairn – review of draft article on IRR  

                                                ii.      Ben: budgeting exercise for grant applications; NHANES – lower female mortality for women taking birth control medications

                                              iii.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.  Also: Prezi demo. 

                                              iv.      Amanda: presentation and interpretation of data in articles

                                                v.      Christina Cruz, 3^rd year FM resident with questionnaire for mild serotonin withdrawal syndrome on 12/6 or 12/13

Recorder: Connie van Eeghen and Kairn Kelley

Clinical Research Oriented Workshop (CROW) Meeting: Oct 24, 2012

Present: Abby Crocker, Kairn Kelley, Amanda Kennedy, Ben Littenberg, Charlie MacLean, Betty Rambur, Connie van Eeghen

1.                  Start Up: A virtual meeting for Amanda and Connie, thanks to Charlie’s technical expertise.     

2.                  Presentation: Betty Rambur shared a concept paper as a precursor to a grant proposal intended for a credentialing organization, American Nurses Credentialing Center, which credentials NPs and hospitals for Magnet status (FAHC is not a Magnet hospital; Dartmouth and Southwestern are).  It is unusual for this organization to provide research grants; this may be part of a strategy to demonstrate the value of Magnet hospitals.  One requirement of Magnet hospitals is that nurses must be involved in research.  This is very difficult for small hospitals.   The topic is about the unintended consequences of measuring quality.  The grant is capped at $1.2m for 3 years, with no more than 30% indirects.

a.       Three phases: development of perceptual measures of unintended consequences, measurement by surveys across New England hospitals, and development of nurse research skills. 

                                                  i.      Survey population: 7,000 active nurses in Vermont; also will include the other states in the region

                                                ii.      Phase 1: developing the metrics (in Magnet hospitals) that identify unintended consequences; using Magnet hospitals only may produce incomplete or biased metrics

                                              iii.      Phase 2: measuring quality usually results in a drop in quality measures at first, as participants start seeing issues in quality previously ignored; these measure the harm that was caused

                                              iv.      Phase 3: develop research capacity in nurses – how strongly should this section be emphasized?  Talk to the grant administrator?

b.      Budget considerations:

                                                  i.      Supports 10 interns, salaries and traveling, interaction costs (i.e. maintaining the relationship with the interns throughout this project)

                                                ii.      Survey administration

c.       Recommendations

                                                  i.      Write more simply, like the elevator pitch that started this meeting

                                                ii.      Are there more direct measures (involving primary data collection or data base analyses) that can be proposed in the concept paper?  Something more than perceptual surveys, e.g. Treatment with antibiotics – something that can be measured objectively via EMR.  There is a specific nursing data set (NDQA?), many elements of which are proxies for staffing: patient falls, pressure ulcers.  Note: accessing primary data in the non-Magnet hospitals could be difficult.

1.      Keep the survey as a metric

2.      Consider metrics connected to the nursing data set

3.      Look at the impact of 30-day mortality rate (due to delayed re-admission)

4.      Hospital acquired infection due to delayed discharge

5.      More immediate indications of harm

6.      The metrics may be the outcome of the development of the student interns

7.      Consider assigning each intern (or 2 interns) a metric for which a protocol is written.  All interns collect on all metrics.

8.      Create a model that starts with latent factors (the perceptual data) and leads to the harm metrics (using Reason’s Latent Factor Model in reverse)

                                              iii.      Consider a systematic review as a first step in the proposal, which can lead to semi-structured points for the focus group to develop the issues

                                              iv.      Consider requesting a lower level funding; have the funder rethink the funding priorities

3.                  Abby’s Update: Current status of her research project

a.       Three possible articles, the first is focused on the predictors of neonatal abstinence syndrome.  Each column is a variable, NAS+ babies, NAS- babies, p value.  Within those columns, Abby has identified the n, missing data, and standard deviation.  Discussion:

                                                  i.      State N overall, and then n for each analysis, and don’t report the “missing data” number

                                                ii.      Univariate variables and descriptors were used to create a predictive model.  The table Abby developed shows where this model came from.  Variables with high p values were removed from the model, one at a time.  The final logistic regression model included 3 variables:

1.      Total dose of methadone of mother per day at the time of delivery (only 100 moms were on methadone) (lower is more protective)

2.      APGAR score (higher is more protective)

3.      Head circumference standardized for estimated gestational age (z score) (higher is more protective)

                                              iii.      This model is not causal, just predictive.  The three variables are all immediately available at time of birth. 

                                              iv.      Consider breaking the table down into several tables, to make interpretation easier for the reader.

                                                v.      Abby also discovered that the babies with earlier birth years (from the beginning of the study) were more likely to be NAS+.  This may be because reporting became more accurate as nurses got more comfortable with this assessment.  Birth year doesn’t have predictive value. 

                                              vi.      Abby also calculated the ROC (Receiver Operator Characteristic curve), which is approximately 67% - so not a great predictive tool.  Ben and Abby came up with a calibration curve, which describes the logistic regression model: for each baby, there is a predictive value of getting NAS (from 0 – 1).  Each baby gets a value on this scale.  The calibration curve assigns a percentage chance of getting NAS by groups of scores.  The predictive ability of the curve should go up as the percentages go up.  The confidence intervals are wide but the slope is 45 degrees – so well calibrated, but not precise. 

1.      Each group (NAS+, NAS-, and all other babies) can be studied independently.

2.      Recursive partition might be appropriate – per past heated discussions.

b.      Discussion continued on the other tables and figures included in this publication and segued into the beauty of APGAR scores.  If we use the ROC as a predictive tool, what is the action rule for an infant with 45% probability?  This may not be discriminative in a clinically relevant range.

4.                  Next Workshop Meeting(s): Thursday, 2:00 p.m. – 3:30 p.m., at Given Courtyard Level 4. 

a.       Nov 1: Kairn: final draft of article on IRR (no Connie, Ben)

b.      Nov 8: Rodger & Connie: presentation on Strategies on Implementing BHI

c.       Nov 15:

d.      Nov 22: THANKSGIVING… take the day off J

e.       Nov 29: Prema: draft grant application

f.       Future agenda to consider:

                                                  i.      Kairn – review of draft article on IRR  

                                                ii.      Ben: budgeting exercise for grant applications; NHANES – lower female mortality for women taking birth control medications

                                              iii.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.  Also: Prezi demo. 

                                              iv.      Amanda: presentation and interpretation of data in articles

                                                v.      Christina Cruz, 3^rd year FM resident with questionnaire for mild serotonin withdrawal syndrome

Recorder: Connie van Eeghen

Clinical Research Oriented Workshop (CROW) Meeting: Oct 11, 2012

Present: Abby Crocker, Kairn Kelley, Rodger Kessler, Ben Littenberg, Prema Menon, Connie van Eeghen

1.                  Start Up: Here’s to IT helpers who need screen shots of “Locked.  You are not authorized to…” in order to believe that authorization is, in fact, denied.   

2.                  Presentation: Rodger & Connie – Aim Statement and Approach sections of R03 grant application for Toolkit for Primary Care Behavioral Health

a.       The group had an extensive and detailed discussion about how to present the design, which is essentially good, in ways that are easier to understand and better supported.  Key ideas:

                                                  i.      Eliminate redundancy and keep simplifying, starting with the Aims page.

                                                ii.      Conceptual diagrams should focus the reader on the key points for this, particular, application.

                                              iii.      Do not indulge in the luxury of stating in the narrative what is in a table, or vice versa. Choose one or the other; the page restrictions make this a necessity.

                                              iv.      Create new sections when logical: “Investigators” and “Intervention,” as examples.

                                                v.      Introduce the overall design in a few simple words. 

                                              vi.      Keep Aim 1 and 2 design elements distinct from each other; they have different kinds of subjects, data collection methods, analyses, etc.

                                            vii.      Every figure or table should have a well-defined question that it is answering and it should answer it as clearly and simply as possible.

                                          viii.      Simplify the analysis to two observations and, in limitations, explain why this is not a limitation.  Adjust for loss to follow up.

b.      Thanks for everyone’s input.  As usual, invaluable. 

3.                  Next Workshop Meeting(s): Thursday, 2:00 p.m. – 3:30 p.m., at Given Courtyard Level 4. 

a.       Oct 18: Christina Cruz, 3^rd year FM resident with questionnaire for mild serotonin withdrawal syndrome

b.      Oct 25: Abby: update (Connie by phone)

c.       Nov 1: Kairn: final draft of article on IRR (no Connie)

d.      Nov 8:

e.       Nov 15:

f.       Nov 22: THANKSGIVING… take the day off J

g.      Nov 29: Prema: draft grant application

h.      Future agenda to consider:

                                                  i.      Kairn – review of draft article on IRR  

                                                ii.      Ben: budgeting exercise for grant applications; NHANES – lower female mortality for women taking birth control medications

                                              iii.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.  Also: Prezi demo. 

                                              iv.      Amanda: presentation and interpretation of data in articles

Recorder: Connie van Eeghen