IRB self-evaluation form also evaluates HRPP performance
Survey can be adapted for other institutions
IRB board member self-evaluations are crucial for determining how members view their IRB service — and measuring the performance of the HRPP itself. But IRB administrators who are looking for self-evaluation tools may have a hard time deciding where to start, or which issues should be the focus.
Enid Virago, PhD, CIP, CCRP, looked to develop a board member self-evaluation tool as a quality improvement project, and as a means to recruit and retain more IRB board members.
"We have four big IRB panels, and we needed to recruit more members," says Virago, QI/QA program manager at the Office of Research and Innovation at Virginia Commonwealth University in Richmond. "We were trying to figure out how to recruit and keep them, and where their educational deficits were."
Virago partnered with Pennsylvania State University colleague Joanna Lyons, RN, DEd, to develop a self-evaluation tool that any IRB office could use. Penn State had been conducting self-evaluations for about five years and developed a survey from a "skeleton" of a form, Lyons says. "Enid and I got together and added more to it," she says. "We have been consistently tweaking it. It’s a work in progress." In addition to wanting to gather more information to help in the recruitment, preparation, and retention of effective board members, "this was also driven by AAHRPP requirements for IRB evaluation, and we wanted something more formal than we had before," she says.
Virago and Lyons developed the 12-page "Evaluation of Research Protections Programs and Committee Membership Self-Evaluation" with questions on expectations and experiences of board members. It contains qualitative, open-ended questions as well as statements that can be checked off as applicable, or ranked from "strongly agree" to "strongly disagree." The survey measures satisfactions, challenges, perceptions of IRB service, quality of IRB education, and time needed to conduct reviews. (For examples of survey questions, see box on page 142.)
Virago and Lyons used the survey as a quality improvement effort at their respective institutions and presented a poster of the results at the AAHRPP annual conference in spring of this year. "About 27 people asked for it [the survey] for their own institutions," Virago says. "The survey can be adapted and sections added that would be useful to other IRBs."
Recently, they filled an additional 45 requests for the survey from other institutions. These institutions include medical teaching hospitals, social behavioral research institutions, military, for-profit, and non-profit national and international organizations.
Study results
When Virago and Lyons analyzed the survey data from their institutions, they were surprised by some of the results: Many IRB board members did not feel knowledgeable enough about the federal regulations.
In addition, many board members reported that they did not feel as comfortable with IRB policies and procedures as they thought they should. "The criteria for approval are the most important thing, but the standard operating procedures can muddy the waters, and it’s not always a clear distinction between what is a clear [institutional] policy and what is regulatory," Lyons says.
One question, Lyons says, is whether board members are interpreting the approval criteria in the regulations in a consistent way. "I think the interpretation part is something that IRB members had questions about," she says.
Institutions interpret the regulations in their own ways and develop policies based on the interpretation, Virago says. Each institution can vary in its interpretations. "When someone arrives at VCU from a different institution, they may see that the regulations are nuanced differently than their previous institution," she says.
Finding time to conduct reviews before IRB meetings also proved to be challenging for board members. "It was very clear that time was an issue" for most board members across both institutions, Lyons says.
"One of the other challenges [at Penn State] is in the documentation of reviews in the electronic system," Lyons adds. "Just navigating the system, whether it’s a new or different one, can be challenging. We moved over to a new one — there are always challenges in learning how to navigate that."
Lyons and Virago also found another surprise: Some of the things the board members found challenging were the same things that the board members also found to be satisfying.
For example, while board members may not have felt as knowledgeable as they wanted to be on federal regulations, they found enjoyment in the learning process. "In contrast to the challenges, they said that learning about ethics and the regulations are very satisfying," Virago says.
"It is interesting that they thought it was a challenge to learn the regulations, but satisfying to get the understanding," Lyons adds. "Even though learning regulations, guidance, and policy is a challenge, it does appear that members really do want to learn."
Board members gained social satisfaction as well, the data showed. "They liked being an IRB expert in their field, and liked meeting new colleagues," Virago says. "They also enjoyed learning about ethics and studies in other fields."
The board members who reviewed predominately social and behavioral research seemed to view serving on the IRB as a commitment to institutional service and compliance, while board members who predominately reviewed clinical trial research enjoyed the social and professional recognition and satisfaction, she adds.
Institutional improvements
Lyons and Virago are using the evaluation data to gain insight to strengthen education programs for their IRB members. For instance, Lyons and the Penn State IRB asked members for educational suggestions. "We’re reinforcing the criteria for approval, and in-meeting education sessions target some of the topics members have mentioned in their evaluations," Lyons says.
VCU is planning to hire an educator, Virago adds.
The IRB also uses targeted education for certain types of studies. For example, if a study involving prisoners is on the meeting agenda, education on the approval of prisoner studies might be presented at the start of the meeting.
"It’s targeted so it’s a just in time’ sort of thing and it happens right before they talk about it," Lyons says. "Even though they have their checklists and worksheets, they have this education that they can apply right away."
Tackling the issue of review time has been a little trickier. Both Penn State and VCU were able to get release time from IRB members’ departments to ensure that at least some members have adequate time to review protocols. VCU IRB members get a stipend and credit toward tenure for their service, Virago says. "However, pay and credit towards tenure doesn’t seem to be a big motivator for the biomedical reviewers, and oftentimes they are too busy to take release time," she says.
In the future, Virago and Lyons hope to share the survey with more IRB programs and update it based on user feedback. "When we send it to people, we ask them to send us comments if they use it," Virago says. "We consider it a living document, and we’d like to see any changes so we can continue to make it better. If they want help writing questions, they can give us a buzz and we can help."
