Clinical Research Oriented Workshop (CROW) Meeting: July 19, 2012

Present: Kairn Kelley, Rodger Kessler, Connie van Eeghen

1.                  Start Up:  Interesting fact from Rodger – when NIMH study section participants finish their review of a grant application that they just finished discussing and make up their minds about the final score, they must self-declare if they are going “out of range,” or giving a score that is higher than or lower than any of the scores given by the in-depth reviewers.  Not many do this, so the “sale” of the application has to be made with those three reviewers in mind.  Why is this relevant?  Connie and Rodger’s application didn’t get to full section review, so all we have to work with are the comments from the three reviewers, which we’re hoping is enough feedback to make it all the way to the finish line.

2.                  Presentation: Connie and Rodger’s R03 Critique

a.       This grant application is being rewritten for resubmission in October.  The one page introduction should address all issues raised by the reviewers and all changes to the draft should be identified by italics or underline. 

b.      Much of the discussion centered on or kept returning to the aims.  Simplify the aims, so that they are easy to read and remember.  Specify the aims really well:

                                                  i.      Aim 1: Make the preliminary toolkit roadworthy so it can be disseminated.

                                                ii.      Aim 2: Confirm that process measures, which is what the toolkit addresses (such as treatment initiation rates, referral rates), are improved.  Characterize this aim as “Reach” and an indication that we should continue to develop and disseminate the toolkit.

                                              iii.      Aim 3: Demonstrate whether outcome measures, which is what BHI addresses, improve using a prospective cohort study.  This study will measure whether patients who received BHI services in the clinic (as developed by the Toolkit) also had changes in symptom scores for depression at 3 and 6 months after screening.  The study will also measure the same data on patients who were screened but, for some reason, did not get BHI services, to determine whether their symptom scores change over time.  Characterize this aim as “Effectiveness.”

                                              iv.      Notes on the aims:

1.      There is not enough space in the aims to explain the mixed methods approach, so don’t.  Explain this in the methods section and expect the researchers to look for it there.

2.      It will also be possible for us to measure utilization of hospital admissions, PCP visits, and ER visits using our source of data.  This may be another perspective on “effectiveness.”  The focus of the grant is on toolkit development, not efficiency, so we had a short debate on whether to include this in the third aim statement and didn’t come to a conclusion on this.  It can certainly be included as work we can do in future studies.

c.       The approach section received the lowest scores from the reviewers and was the focus of the remainder of the discussion.  We focused on the first comment and came up with the following plans:

                                                  i.      State a model that links the toolkit to BHI integration to care.  Use Peak’s 3 world model to support the need for alignment of operations with financial and clinical processes. 

1.      Example statements: The success of BHI is determined by its operational implementation.  The purpose of the toolkit is to provide the operational details that lead to successful implementation and a strategy to customize each operational element to the individual practice. 

2.      Cite Gawande’s WHO article to support the use of a checklist in refining clinical practices by using operational processes.  Also, check Rodger’s application for the Simulation Lab for references related to the relationship between operations and clinical outcomes. 

                                                ii.      Clarify that this study is providing an initial test of the previously developed toolkit to confirm its acceptability (refer to the TAM model) and to improve it as needed.  While the method has already been developed, the toolkit (manual, checklists, examples, etc.) has not been developed.  The value of doing this will be measured in clinical outcomes and, later, in efficient use of health care resources.  

d.      Next steps: Connie to send draft to Rodger by Monday morning.

3.                  Workshop Goals for 2012:

a.       Journal club: identify UVM guests and articles; invite to CROW ahead of time

b.      Research updates: share work-in-process

4.                  Next Workshop Meeting(s): Thursday, 11:00 p.m. – 12:00 p.m., at Given Courtyard Level 4. 

a.       July 26: Connie: R03 draft (no Ben)

b.      Aug 2: Abby – “How we picked the predictive model for the NAS article”

c.       Aug 9: (no Abby)

d.      Aug 16: (no Abby)

e.       Aug 23:

f.       Aug 30: (new schedule?)

g.      Future agenda to consider:

                                                  i.      Ben: budgeting exercise for grant applications

                                                ii.      Journal Club: “Methods and metrics challenges of delivery-system research,” Alexander and Hearld, March 2012 (for later in the year?).  UVM authors who have published interesting design articles (Kim, Osler)

                                              iii.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.

                                              iv.      Amanda: presentation and interpretation of data in articles

                                                v.      Sharon Henry: article by Cleland, Thoracic Spine Manipulation, Physical Therapy 2007

Recorder: Connie van Eeghen

Clinical Research Oriented Workshop (CROW) Meeting: July 5, 2012

Present: Abby Crocker, Amanda Kennedy, Rodger Kessler, Ben Littenberg, Connie van Eeghen

1.                  Start Up:  

a.       What are the predictors of successful collaboration within a practice?  It’s more than excitement and clearly includes a champion.  To be continued…

b.      New CTS applicant: surgical resident (Charles Parsons) to start today; will attend the Design class starting this afternoon

c.       Questions about opportunities from COM:

                                                  i.      Laurie St. Gelais: yes, a good opportunity and further discussion is warranted but I deleted the email and can’t for the life of me remember what it is about.  Vacation brain syndrome… sorry!  Would someone please come to my rescue?

                                                ii.      Hilda Alajajian: Emerging Researcher NIH training opportunity to get an insider’s view of  NIH study section in the Early Career Review Program.  Can be done through training on website. 

                                              iii.      Ruth Farrell: HHS has new conflict of interest regulations.  Information sessions are on Monday, July 9th, 3:30 p.m., Carpenter Auditorium and Wednesday, August 1st, 9:00 am - Carpenter Auditorium.  Go if you want to go.

2.                  Presentation: Abby is working on research articles this summer, one in particular, including the use of STATA.  She is looking ahead at her plan for the next few weeks of work.  Goal for today: skeleton of the paper and what to move forward without a complete analysis. 

a.       Check out the “Report in Clinical Usage” book

b.      Goal: understand the predictors for neonatal abstinence syndrome that indicate follow up.  Essentially: build a model

c.       Target publication: Pediatrics.  Use the instructions on this web-site to draft the article.

d.      Title: A Predictive Model for Neonatal Abstinence Syndrome

e.       Background: The 50,000 foot hook (with a number) of why important.  What NAS is and how identified.  Why the current identification method is sub-optimal.  The need for better prediction.   “Therefore, we sought to… “ or “Therefore, the purpose of the study is to…”  Shoot for three paragraphs.

f.       Methods: Description of design, with statement of IRB approval. (This is the place to start, in writing the article)

                                                  i.      This is a secondary analysis of an existing data set. 

                                                ii.      Describe the data set as a prospective cohort of women, etc.  “To develop our model, we took advantage of an existing data set of pregnant women created for the purpose of…  The data were collected through the ICON clinical registry that is a QA registry that includes (# live births for # delivering women)”  “Includes all deliveries between 2005 - 2012 of opiate exposed women at FAHC…”  “Prospectively collected, with variables including social data about mother, clinical data about pregnancy, delivery, and immediate post-partum period…”  

                                              iii.      Subject selected (everybody?) with inclusion/exclusion, e.g. mothers not in treatment for substance abuse.  Refer to 1^st table here?  Decide later.  “All women in database were potentially included, but we excluded those that (died, didn’t have a chance to develop NAS)”  IRB statement follows this description.

                                              iv.      Variables

1.      Outcomes

2.      Predictors (there are many; may need to use a table).  Important to identify the starting point of the study: what did you consider.

a.       Definitions: what is substance abuse?  Defined by opiate dose?  Use morphine equivalence?  Ask Amanda.

                                                v.      Analytic approach.   

1.      Descriptive statistics: how narcotics were categorized.  Use of means and standard deviations for total population of eligible women for this study (770); medians and quartiles; proportions...  Box plots…  Use of morphine equivalence can also go here. 

2.      Univariate analysis: Predicting a binary outcome (with and w/out NAS): continuous (t test if it’s normally distributed or use rank based test if not normally distributed) and dichotomous (Chi square, exact test…)  Each predictor was analyzed separately with respect to the outcome.

a.       Purpose: test each predictor vs. NAS

b.      Method: x methods for this analysis

c.       Table 1: variables by total with P value in last column

                                              vi.      Model building process: P value <=0.10.  Binary outcomes can be predicted by one of the following methods: logistic regression, recursive partitioning, and neural networks. 

1.      Logistical Regression used when all sub-groups are equally predictive, i.e. age of mother is as important a predictor in male vs. female babies.

2.      Ben strongly recommends one of the others.  To be continued…

g.      Results (the rest of the outline will be continued at a later date)

h.      Discussion & Limitations

i.        Conclusion

j.        Next step:

                                                  i.      Get organized around the model building process.  Meeting with Ben tomorrow.

1.      How many variables can this study tolerate before “over-fitting” the model?  There are 770 subjects in the data base.  About 15.  (Over-fitting: fitting random noise into the model.  There are ways to minimize this noise, but they have trade-offs.)

                                                ii.      Abby will get editing support from Amanda.

3.                  Workshop Goals for 2012:

a.       Journal club: identify UVM guests and articles; invite to CROW ahead of time

b.      Research updates: share work-in-process

4.                  Next Workshop Meeting(s): Thursday, 11:00 p.m. – 12:00 p.m., at Given Courtyard Level 4. 

a.       July 12: Rodger – outline of new research paper (Connie by phone, no Abby or Kairn)

b.      July 19: (no Abby or Ben)

c.       July 26: (no Ben)

d.      Aug 2: Abby – “How we picked the predictive model for the NAS article”

e.       Aug 9: (no Abby)

f.       Aug 16: (no Abby)

g.      Aug 23:

h.      Aug 30: (new schedule?)

i.        Future agenda to consider:

                                                  i.      Ben: budgeting exercise for grant applications

                                                ii.      Journal Club: “Methods and metrics challenges of delivery-system research,” Alexander and Hearld, March 2012 (for later in the year?)

                                              iii.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.

                                              iv.      Amanda: presentation and interpretation of data in articles

                                                v.      Sharon Henry: article by Cleland, Thoracic Spine Manipulation, Physical Therapy 2007

Recorder: Connie van Eeghen

Clinical Research Oriented Workshop (CROW) Meeting: June 28, 2012

Present: Kairn Kelley (by phone), Rodger Kessler, Connie van Eeghen

1.                  Start Up:  Affordable Care Act passed – we will have a better battle ahead of us.

2.                  Presentation: Rodger and Connie shared the critique received they on their R03 application, which did not get scored well enough for an in-depth review.  The agenda for this CROW session will be focused only on comments related to “Significance” due to a conflict in Rodger’s schedule. Other topics will be discussed in future meetings; comments are welcome from all. 

a.       Weakness: Link of workflow implications to patient outcomes, conceptually, not developed

                                                  i.      Discussion: Connie found no existing model in the literature. Her dissertation showed a relationship (association, not causal) between clinician perception of processes and clinician perception of clinical outcomes. The Helfrich model proposed determinants of implementation effectiveness which were ultimately linked to measures of success, but this model is not specific to workflow (i.e. Lean) or to patient outcomes.

                                                ii.      Strategy: Look to following sources for rationale and associations between process and clinical outcomes: Sarah Scholle (NCQA VP of Research and Analysis) regarding survey data on associating processes and clinical outcomes and Katon & Unutzer’s body of work from the MN Collaborative Care experiences (AHRQ 2008 evidence report on collaborative care and MH outcomes).

b.      Weakness: Outcome Variables of specific Aims are process based rather than clinical.  Possible Strategies:

                                                  i.      Add initial and follow-up patient screening scores to the outcome measures

1.      Screening measures exist at the clinic level and can be used in before/after comparisons

2.      Unclear if GAD or PHQ are accessible and used consistently in the process

3.      Outside the scope of the project to get patient contact data

4.      Should it be the primary measure?

a.       Not clear (some sites will be using the screening but will not use panel based management or there won’t be an onsite clinician)

b.      How could/should differences in 6 month change scores be linked back to the toolkit?

c.       Should we have a comparison base? We’ll have the data base, so it’s not a large task to include other practices.  However, based on a two site pilot study, we will not have the power necessary to establish relationships.

5.      Are we changing the MISSION of this project?

a.       Toolkit is a solution to help practices change

b.      Not trying to test if PCBH makes a difference in screening outcomes.

c.       We would have to add practices that have PCBH but don’t have the toolkit intervention.

d.      Outcome: reduced costs, happier staff, PRESUME patients will have better outcomes. Staff satisfaction is an example of an outcome of successful implementation, which is the goal of the toolkit. 

e.       How do you get to demonstrating clinical outcomes to be consistent with the mission and responsive to the critique?

                                                ii.      Make it clear that the primary outcome is process based: treatment initiation

                                              iii.      Also trying to put clinical outcomes in place for future evaluation and an important part of future research.  In doing so, the toolkit should include measurement of clinical outcomes. 

c.       Rodger and Connie will continue to work on these and other weaknesses.  Part of their work is to develop a strategy that makes it easy for reviewers to see how their comments were responded to. 

3.                  Workshop Goals for 2012:

a.       Journal club: identify UVM guests and articles; invite to CROW ahead of time

b.      Research updates: share work-in-process

4.                  Next Workshop Meeting(s): Thursday, 11:00 p.m. – 12:00 p.m., at Given Courtyard Level 4. 

a.       July 5: Abby – research update (no Kairn)

b.      July 12: (Connie by phone, no Abby or Kairn)

c.       July 19: (no Abby or Ben)

d.      July 26: (no Ben)

e.       Aug 2:

f.       Aug 9: (no Abby)

g.      Aug 16: (no Abby)

h.      Aug 23:

i.        Aug 30: (new schedule?)

j.        Future agenda to consider:

                                                  i.      Ben: budgeting exercise for grant applications

                                                ii.      Ben: Writer’s workshop on the effect of the built environment on BMI (Littenberg & Austin Troy)

                                              iii.      Journal Club: “Methods and metrics challenges of delivery-system research,” Alexander and Hearld, March 2012 (for later in the year?)

                                              iv.      Rodger: Mixed methods article; article on Behavior’s Influence on Medical Conditions (unpublished); drug company funding.  Also: discuss design for PCBH clinical and cost research.

                                                v.      Amanda: presentation and interpretation of data in articles

                                              vi.      Sharon Henry: article by Cleland, Thoracic Spine Manipulation, Physical Therapy 2007

Recorder: Kairn Kelley