Any researcher collecting data from web-based surveys worries about data integrity issues because of the well-established problem of people giving false information to get offered incentives.1,2 Clicking through a quick demographic survey to get a gift card is one thing, but volunteering for a lengthy interview under false pretenses would seem unlikely. Yet multiple researchers encountered this type of fraud while conducting qualitative studies.
When Elizabeth Mizerek, PhD, RN, FN-CSA, CEN, CPEN, CNE, FAEN, was recruiting nurses for her dissertation study, she was pleased to see that about 40 people had completed the demographic screening tool in just one week. “There were so many participants, I thought that the topic was really resonating with people,” says Mizerek, director of nursing education at Mercer County Community College in West Windsor, NJ. Many individuals took the next step, setting up a time to conduct an interview via Zoom. “There were no red flags, initially, to think it was anything other than legitimate research participants,” recalls Mizerek.
Mizerek started out with some open-ended questions using a semi-structured interview format, but the answers given by the “nurses” seemed strange. When asked what triage system the department used, one participant erupted in a coughing fit, went off screen to get a glass of water, and came back with a response (presumably, having Googled the answer). When Mizerek followed up with a more in-depth question (on how a patient with acute stroke would be triaged), the call was suddenly disconnected. Other interviewees gave non-responses to key questions, so Mizerek informed them they did not meet the study criteria.
Mizerek’s emergency nursing background made it easy to detect the fraud. As a result, the data set was not contaminated. “But what if I wasn’t able to identify it before it was too late? That’s where it’s really concerning,” says Mizerek.
Ultimately, Mizerek completed the study using a targeted recruitment approach, such as through a Facebook group run by an emergency nurse colleague that was populated by other known emergency nurses.3 To reach geographically diverse participants, Mizerek reached out to emergency nurses she knew of personally in specific regions, drawing on contacts she had connected with over many years though the Emergency Nurses Association. “It’s easier when you have a genuine connection with real-life people that you know, who can introduce you to real-life people who they know,” says Mizerek.
Lisa A. Wolf, PhD, RN, CEN, FAEN, FAAN, had a similar experience when recruiting for a qualitative study on emergency nurses’ views on obstetric emergencies.4 Wolf circulated a link for recruitment, noting that a $50 gift card would be given to participants who completed a focus group. Very quickly, about 300 people signed up. Researchers arranged for two groups of 40 participants to meet up in Texas, and arrived at the conference room at the appointed time. “And nobody showed up,” says Wolf, director of emergency nursing research for the Emergency Nurses Association and associate professor at UMass Amherst. At first, the situation made no sense. As is typical in qualitative studies, research staff had over-recruited, knowing that some people inevitably forget or change their minds. While the researchers waited for participants to arrive, they started to investigate further. Many of the emails indicated a six-hour time difference. The demographics also didn’t add up, such as 21-year-olds who claimed to be nursing managers.
To try to salvage the situation, the researchers managed to arrange Zoom calls with some of the people who had signed up. Individuals who claimed to be emergency nurses were seated in a bare room with a single lightbulb in the dark — yet it was the middle of the day. None could answer basic questions about emergency nursing. The fake participants demanded to be paid for their time. Fortunately, the consent form stated that only participants who met the criteria would receive gift cards. However, considerable funds had been used for researchers’ travel and lodging. “It was a complete waste of time and resources,” says Wolf. To salvage the study, the researchers successfully reached out to contacts in the state to arrange in-person interviews with actual emergency nurses.
Afterward, the researchers reflected on these unsettling experiences. Several had performed qualitative research for many years, and never had anyone sign up under false pretenses. The group concluded it had to be the gift cards that attracted the fraud. Some colleagues even reported that institutional review boards had required fake “participants” to be paid. “I had no idea it was this prevalent. It never occurred to me,” says Wolf. “But there is a whole industry of people who take surveys for money. I suspect these people are in developing countries, where a $50 gift card is a lot of money.”
Wolf, Mizerek, and colleagues conducted a literature search and found multiple papers on survey data integrity, but nothing on data integrity concerns with qualitative studies. “I was pretty surprised that people would think they are going to fake their way through an hour-long interview on a topic about which they know nothing,” says Wolf. The researchers authored a paper to offer insights on how to prevent fraud from occurring in future studies.5 “Recruiting just needs to be a little more thoughtful,” says Wolf. “You can’t just put stuff out on the Internet and expect to get valid responses.” Some recommendations to prevent fraud during study recruitment include:
• Reach out to a network of contacts, or post information in carefully selected social media groups comprised of the study populations you are looking for.
• Avoid incentives altogether if possible; or, if incentives are necessary, offer a raffle for a single gift card.
• Offer incentives later in the screening process, only after verifying that the participant actually meets the criteria.
• Ask screening questions that require a free-text response — and that only someone from the targeted population would be able to answer correctly.
• Take advantage of the full capabilities of software.
Settings may need to be adjusted to flag people who are completing a survey multiple times (indicating they are figuring out how to respond to qualify) or multiple responses from one IP address. “If you look at your responses, you might see they are all bots,” says Wolf. “The software can’t stop them from coming through, but it is getting better at identifying it. These companies are equally invested in maintaining data integrity.”
Researchers need to be vigilant at removing suspicious responses from study samples. Some researchers need to publish a certain number of papers to get tenure. A resident may need to complete a project based on survey data in a certain time frame. “The pressure to get it done may be more pressing than to do a really good study,” warns Wolf. “But if we’re not doing high-quality work, then, really, there’s no point. And it’s an ethical violation to take data you know are not valid and use it anyway.”
- Panesar P, Mayo SJ. “Taking out the trash”: Strategies for preventing and managing fraudulent data in web-surveys. Can Oncol Nurs J 2023;33:283-284.
- Levi R, Ridberg R, Akers M, Seligman H. Survey fraud and the integrity of web-based survey research. Am J Health Promot 2022;36:18-20.
- Mizerek EA. Exploring clinical judgment and influences on emergency nurses’ handoff process. Widener University ProQuest Dissertations Publishing, 2023.
- Wolf L, Noblewolf HS, Callihan M, Moon MD. What if it were me? A qualitative exploratory study of emergency nurses’ clinical decision making related to obstetrical emergencies in the context of a post-Roe environment. J Emerg Nurs 2023;49:714-723.
- Mizerek E, Wolf L, Moon MD. Identifying and mitigating fraud when using social media for research recruitment. J Emerg Nurs 2023;49:530-533.
