Evaluating the Development, Implementation and Dissemination of a Multisite Card Study in the WWAMI Region Practice and Research Network
Abstract
Background
Practice-based research networks (PBRNs) promote the conduct of research in real-world settings by engaging primary care clinicians as champion research collaborators. Card studies are brief surveys administered to patients or clinicians at the point of care. The objective of this paper is to describe the design and evaluation of a card study methodology that the WWAMI Region Practice and Research Network (WPRN) used to develop research partnerships across multiple member sites.
Methods
We used a collaborative model to develop, implement and disseminate the results of a network-wide card study to assess patient preferences for weight loss in primary care. After the card study data collection was completed, we conducted individual and focus group interviews and a brief survey of participating practice champions.
Results
Increased research engagement and personal and professional developments were the primary motivators for participating in the development of the card study. Increasing research activity at practices and learning information about patients were motivators for implementing the study. Their participation resulted in champions reporting increased confidence in collaborating on research projects as well as the development of new clinical services for patients.
Discussion
This collaborative model positively influenced research capacity in the WPRN and may be a useful strategy for helping PBRNs conducted translational research.
Introduction
Practice-based research networks (PBRNs) promote the conduct of research in real-world settings by bringing together academic investigators and primary care clinicians.1-6 To achieve successful research partnerships, PBRNs must engage primary care clinicians as research collaborators and foster the research capacity of primary care practices.7, 8 The WWAMI (Washington, Wyoming, Alaska, Montana, and Idaho) Region Practice and Research Network (WPRN) used a card study methodology across multiple member sites to develop research partnerships and engage primary care clinicians in research.
Card studies are a well-established method for conducting observational studies in PBRNs.9 A card study is a type of survey designed to gather data at the point of care.9 Clinicians or patients complete a brief survey (typically the length of an index card) in primary care practices.9 Card studies have been widely used in PBRNs because of their feasibility and ability to answer important clinical questions.9 Card studies have demonstrated superior accuracy, compared to medical record review, in identifying certain clinical conditions, such as spontaneous abortions.10 The WPRN created a new model to foster engagement of primary care research champions by involving them in the development, implementation and evaluation of a multisite card study, “Patient Preferences for Weight Loss in Primary Care.” The University of Washington based WPRN Coordinating Center led an evaluation of this model to assess the practicality of this approach for engaging primary care clinicians in research and building the research capacity of practices in the WPRN.
The objectives of this paper are to describe the WPRN's innovative collaborative model for developing, implementing and disseminating results of a multisite card study, and to assess the influence of these activities on primary care clinician research engagement and primary care practice research capacity.
Methods
Setting
The WPRN includes more than 50 clinic primary care practices across five Northwestern U.S. states. To engage practice research champions and build research capacity, the WPRN Coordinating Center regularly develops and implements network-wide projects.7 Practice champions join in the development of these projects based on their clinical interests. The WPRN Coordinating Center provides research support and guidance through academic faculty and research scientist effort. In this study, we used a collaborative approach to develop, implement, disseminate and evaluate the results of the “Patient Preferences for Weight Loss in Primary Care” study, which aimed to determine the degree to which overweight and obese patients are willing to participate in evidence-based weight loss programs.
Development
The WPRN Coordinating Center launched this collaborative engagement project in March 2012. The objectives of the project were to: (1) involve all interested member practices in a new research project, (2) address a topic of importance to member practices, (3) employ feasible, low cost research methods, (4) provide practice- and network-wide feedback to participating practices about their patient populations, and (5) generate interest and enthusiasm for future research in the WPRN.
A card study was selected as a feasible, low cost methodology that would allow many practices in the WPRN to participate.9 After introducing the concept at an in-person network-wide annual meeting, the WPRN Coordinating Center invited all practice champions to suggest topic ideas and clinical questions via email and a conference call, and then joined with interested practice champions to create a small group to develop the study. Five practice champions joined the small group.
Through regular teleconference meetings (eight one-hour meetings over 15 months), the card study development group, supported by the WPRN Coordinating Center, selected the topic of obesity. The topic was selected by popular vote from a list of clinical topics generated by interested practice champions. Other topics the group considered included tobacco use and medication management. The card study development group brainstormed potential scientific questions. The final questions, assessing patient preferences for weight loss in primary care, were based on recently published guidelines from the U.S. Preventive Services Task Force recommending primary care providers screen adult patients for obesity and offer or refer obese patients for comprehensive weight loss treatment. The card study development group then drafted and refined the study instrument, developed implementation procedures, reviewed preliminary results, and reviewed and co-authored manuscripts and presentations (Table 1). In addition, group members received regular email communication from the Coordinating Center with requests to review materials and provide detailed feedback.
| WPRN coordinating center role | Practice champion role | |
|---|---|---|
| Topic selection |
|
|
| Scientific question |
|
|
| Survey instrument |
|
|
| Implementation |
|
|
| Data management |
|
|
| Results dissemination |
|
|
The final version of the card study instrument was a one-page questionnaire with 13 questions. The questionnaire assessed patient self-reported demographics (age, gender, race/ethnicity and insurance status), health (height, weight, overall health), desire to lose weight, reported likelihood of participating in a comprehensive weight loss program and identification of factors important in helping patients decide about participation. The questionnaire was designed to be offered to all English and Spanish speaking, adult, nonpregnant patients checking in for an office visit at each participating practice over a 2-week period.
After completing the card study development phase, which took approximately one year, all WPRN member practices were invited to participate in the study through a presentation at the next in-person annual meeting. Each participating practice identified a practice champion to lead implementation and coordination of the card study at their site.
Implementation
The WPRN Coordinating Center provided all materials for study implementation, including a written implementation guide, printed questionnaires and return boxes. The written implementation guide outlined eligibility criteria, data collection procedures and frequently asked questions. All but one practice completed data collection within a specified one-month window. Staff from the WPRN Coordinating Center attempted to complete a check-in telephone call with each site champion during the first week of data collection and five of 12 champions were reached. All completed questionnaires were returned to the WPRN Coordinating Center by mail, which oversaw data entry and analysis.
Dissemination of results
In addition to analyzing data across sites for peer-reviewed publications, the WPRN Coordinating Center created one-page summaries of both network-wide and practice-specific study results. The WPRN Coordinating Center also provided each site with a spreadsheet file containing their site-specific results. Network-wide results were shared in person at the next WPRN annual meeting and individual and network wide results were distributed electronically to each champion who led implementation of the study at his/her site. This dissemination strategy aimed to ensure that champions could share practice-specific feedback with their staff in a timely and efficient manner.
Evaluation of the WPRN card study process
The card study evaluation mirrored the research project's three key phases: development, implementation and dissemination. Subjects included the 12 practice champions that facilitated implementation of the card study at their sites, five of whom also participated in the card study development process.
To evaluate the small group development process, we invited the five practice champions who participated in the card study development group to join one of two small focus group interviews, each lasting 45–60 minutes. To evaluate the card study implementation process we invited the site champions representing the 12 participating practices to participate in individual telephone interviews, each lasting 15–30 minutes.
The interviews for the development and implementation evaluations assessed practice champions’ motivations to join the project, expectations for participation and effect of completing the card study on future research willingness and capacity. The interviews evaluating the implementation also assessed the process for study approval at each site, practice staff involvement and roles, staff training, and study procedures and monitoring.
The focus group and individual interviews were audio-recorded while the interviewer took detailed notes. Interview notes were reviewed by the principal investigator (AC) and supplemented with information from the audio-recordings for clarity and thoroughness. Notes from each interview were reviewed and coded for key concepts by two independent coders (AC and GK). The research team (AC, GK, and LMB) reviewed all coded transcripts, and discrepancies were resolved by consensus. The principal investigator organized the concepts into overarching themes based on the objectives of the study (motivation to participate, research capacity, successes and challenges) and presented them for review and discussion to the research team.
To evaluate dissemination of study research, the WPRN Coordinating Center conducted a brief electronic survey of the 12 practice champions who facilitated implementation of the card study at their site. The survey questions assessed whether practice champions had shared study results with their sites, what methods of sharing had been used, how results had been received and what influence there had been on the practice or on future research. The anonymous questionnaire was administered securely through a REDCap electronic data capture tool.11 Summary statistics (means and proportions) were calculated for all quantitative responses; qualitative responses were reviewed and grouped into categories. All study procedures were reviewed and approved by the University of Washington Institutional Review Board.
Results
Card study development process
Four of the five practice champions from the card study development group participated in focus group interviews. The practice champions cited a desire to engage in a collaborative project with other WPRN practice champions (Table 2). The opportunity for personal and professional development as well as the possibility of authorship on a publication was also an important motivator. Because participating in the card study development group could represent a significant time commitment for practicing clinicians, we were interested in identifying factors that supported practice champions’ ability to participate. Champions identified WPRN Coordinating Center support as critical to their involvement. Finally, following participation in the card study development group, champions reported additional plans for future research and a greater understanding of card study methods.
| Motivations to participate | |
|---|---|
| Theme | Illustrative quotes |
| Engage in collaborative project with other practice champions in the WPRN | “I wanted to be part of a team, collaborate with others and learn more about how to do card studies and work together.” Practice champion 1 |
| Personal and professional development | “The goal of this is that it could lead to a possible publication as well. I was interested in could I be involved in authorship as well.” Practice champion 2 |
| Supporting factors to participation | |
| WPRN Coordinating Center support | “The [WPRN] coordinators kept it moving and solicited input from us a lot.” Practice champion 1 |
| Impact of small group participation on research skills and future research plans | |
| Increased plans for future research and greater understanding of card study methods | “[Participating] made me want to collaborate more because it was a good experience. Got to see how a card study is developed and how you do the process and learn from the process and how much data you get back when it is implemented well. I've taken that and I'm going to do a card study… Professionally this was a good experience to foster growth in research abilities and desires.” Practice champion 3 |
Card study implementation process
Eleven of the 12 participating practice champions completed individual interviews that are presented as Table 3. Champions cited three major factors influencing their clinics’ decisions to participate in the study: importance of the study topic, feasibility of study methods, and opportunity to increase research participation.
| Theme | Illustrative quotes |
|---|---|
| Factors influencing clinics’ decisions to participate | “Obesity is a significant area of concern for many of our patients. The [card study] approach was a great way to get information on the population we serve and their interest in programs for weight loss.” Practice champion 2 “We were interested in increasing research in the clinic. It is important to us to do research.” Practice champion 4 |
| Factors influencing practice champions’ decisions to lead study at their sites | “I'm the WPRN champion. I'm the person that has that role in the residency. There was not really any discussion of anyone else leading. Just part of my job.” Practice champion 4 |
| Approval process | “The one thing we did early on when we were looking at opportunities, we established a research committee; myself, pharmacy resident, program director, physicians and a resident. We developed the committee, then we filtered ideas through the committee to see if we were willing and interested in participating and timing wise if it fit within the rest of the clinic.” Practice champion 4 “I talked with the program director, clinic manager and clinic associate director. No issues came up.” Practice champion 1 |
| Champion role in implementation | “I was involved in getting information from the [WPRN] Coordinating Center and developing a plan for implementing it in the clinic. Then I worked with the front desk staff and the practice manager that supervises them and we developed a plan. I trained them for a strategy for collecting data that I outlined.” Practice champion 4 |
| Staff training | “You know, maybe it would have been better if I had done some training but I just explained who were the eligible people and guaranteeing that it was anonymous and where to put the results.” Practice champion 3 “I worked with front desk staff and the practice manager that supervises them and we developed a plan. I trained them for the strategy for collecting data that I outlined.” Practice champion 6 |
| Staff motivation | “I made a fun little flyer every morning with pictures on it and a cute or encouraging quote and then I would put how many I had got the day before and how many we had total and how many we still had to go… They liked that. I taped it to an area on their desks that no one can see and taped it at each station and personalized it for each front desk. They felt more connected to me and understood that it was important.” Practice champion 6 “When I discussed with the clinic manager how to incentivize the clinic staff to participate, the clinic manager's response was that this was their job and we don't need to incentivize them.” Practice champion 8 |
| Positive staff reaction to participation | “People were interested in hearing about our results and what we would do with the results. It was neat getting everyone more excited about research and how it might improve patient care.” Practice champion 7 |
| Negative staff reaction to participation | “A few staff felt that it was a burden, another thing to remember and just an extra thing to do.” Practice champion 1 |
| Keys to success | “It takes someone willing to make sure it is going… I don't know how a site could do it without having someone that was motivated and interested and followed through.” Practice champion 6 “I think the easiness of the instrument was very important.” Practice champion 4 |
We sought to understand what factors facilitated individuals’ desire to serve as site champions for the project. Champions believed that their choice to lead the study was a natural fit, given their identification as a WPRN research representative. As local leaders for the study, each champion was required to engage with his/her site leadership to gain approval for participation. In addition to discussions with medical and administrative leaders, one practice champion described a site research committee that had a formalized process for approval.
Practice champions were responsible for operationalizing implementation, including: introducing the study, training, and motivating staff, and monitoring data collection. The intensity with which champions implemented and monitored the study varied. To introduce the study to the practice, champions presented the study to providers and/or staff at an in-person meeting. To train staff on study procedures, champions discussed the study procedures individually with practice staff, though in some cases, champions provided little or no training.
Practice champions felt the work of the study fell within the routine workflow of the practice and expected duties of practice staff. Strategies for monitoring data collection included daily check-ins with front desk staff, providing encouragement, and monitoring the survey return box. Practice champions made efforts to motivate practice staff to participate in the study, such as providing encouragement and feedback.
Practice champions reported a positive response from colleagues and staff about the study. However, three champions reported negative staff responses. In these cases, staff expressed concerns about burdening patients with additional forms or the potential effect of the study on practice workflow (Table 3). Nonetheless, champions reported feeling confident that their practice could undertake future research projects. Practice champions felt rewarded by getting results back about the study.
Card study results dissemination
Nine of the 12 practices responded to the dissemination survey. Eight practice champions reported having used at least one dissemination method to share the results of the study with their sites. Five practice champions shared results at a practice meeting, three posted a hard copy of the one page summary of results in the practice, two sent the one page summary to staff electronically, and one posted the one page study summary on a bulletin board in the waiting room. Eight practice champions reported sharing results with providers at their sites, but rarely reported sharing results with administrative leadership or front desk staff. Seven practice champions reported that the one-page summary of the study results was very valuable and five reported the raw dataset provided to each champion was useful.
When asked what response the practice champions expected from sharing study results at their practice, practice champions cited either an increased interest in research participation or growth or support of clinical services for obesity. The actual responses to their sharing the study results included: pride in participating, and better understanding of patient population needs. Participation in the card study led to development of two quality improvement projects, one research project and two new clinical programs.
Discussion
The ecology of medical care demonstrates that in one month, less than 1% of the US adult population is hospitalized in an academic health center, whereas 11% visit a primary care provider.12 Primary care PBRNs are critical infrastructures for facilitating the conduct of research in real-world settings and play an important role in facilitating research in locations where the majority of patients receive care. A 2007 national survey of PBRNs reported that 18% of PBRNs identified clinical practices’ lack of research experience as a major challenge.13 Use of a collaborative model to develop, implement and disseminate results of a card study in the WPRN successfully engaged practice champions in a network-wide research project and increased some elements of reported research capacity in participating practices. Practice champions who participated in the card study development group reported increased confidence in developing independent and collaborative research projects in the future.
Even with a detailed implementation guide, we found variation in implementation of the card study across participating sites. In particular, there were differences in the level of staff engagement, the amount of training and monitoring provided, and whether incentives for staff were used to encourage participation. Successful strategies to improve research protocol implementation across multiple sites in PBRNs include the provision of face-to-face training, detailed training manuals, and shortening time between training and implementation to maintain study momentum.14, 15 Provision of detailed instructions and materials to facilitate practice champions introducing the study to their practices, training staff and monitoring data collection might have been able to reduce the implementation variation across sites in this study. Our findings confirm that PBRN Coordinating Centers may be a key resource to provide this type of support for multisite research studies in PBRNs.16
A key step in speeding the translation of discovery into practice is facilitating dissemination of scientific evidence into real world clinical practice. In this project, we created a one-page summary of study results that champions used to share results with their practices. This simple dissemination tool was important in ensuring the results of PBRN research reached practicing clinicians, as traditional dissemination strategies such as peer-reviewed publication may take years to change practice.17-19 Participating practices were able to use information learned from the study to develop two quality improvement projects, one research project, and two new clinical programs, suggesting that research participation and active dissemination of research findings were effective for building research capacity and fostering innovative clinical care. Evaluating our participatory process contributes new knowledge to the science of evaluating engagement in PBRNs. This science is key to understanding the best methods to support ongoing engagement and capacity building in PBRNs.
A major challenge for the WPRN Coordinating Center was lack of external funding to support this engagement project. Funding to support PBRN infrastructure remains a challenge for many PBRNs.20-22 Our experience suggests that building research capacity in PBRNs is even more resource intensive than maintenance of PBRN infrastructure. Partnerships with Clinical Translational Science Award (CTSA) institutions, such as the WPRN's partnership with the Institute of Translational Health Sciences, may be an effective method for ensuring more stable infrastructure funding and expanding engagement and capacity building efforts in PBRNs.23, 24
There are several limitations to this study. All participants in the development group were also faculty at community-based Family Medicine Residency programs affiliated with the University of Washington. We did not directly assess practice champions previous research experience. Two practice champions involved in the card study development process were experience practice champions who had worked closely with the WPRN in previous projects, while three had little or no previous research experience with the WPRN. The degree to which the small group process could be implemented successfully among nonfaculty practice champions is uncertain, and less time intensive engagement strategies may need to be developed. Practices without a teaching mission may be more likely to participate in research activities that align closely with clinical or operational activities.24 Focusing PBRN practice engagement work on primary care relevant topics and methods will be critical.
While card studies are a commonly employed method for research in PBRNs, the collaborative model that the WPRN used to develop, implement and disseminate results of a network wide card study successfully engaged a large number or practices in a research project and increased reported research capacity of participating practices. We are not aware of any previously published reports describing this type of collaborative approach to development, implementation and dissemination of research studies in a PBRN. Our rigorous evaluation of the collaborative model the WPRN used sheds light on the challenges and successes inherent in this approach. This collaborative model of research development and implementation provides an effective method to increase stakeholder engagement in clinical research. However, given the resource intensive nature of engagement projects in PBRNs, sustainable models of funding to support PBRN engagement activities are needed if PBRNs are to succeed in speeding the translation of discovery into practice.
Acknowledgments
This project was supported by the National Center For Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR000423. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
