Designing research studies during the pandemic became a challenge for a lot of people in the research community. Qualitative research in particular, involves closer observation and interactions which may be harder to translate remotely. The added challenge during the pandemic is that most potential research participants are already in a Zoom-burnout, and now more than ever, researchers need to get creative in finding data collection methods that are kind to the participants and the research team.
Here are some ideas for conducting qualitative research remotely. Based on my personal experiences, I'm sharing examples from the Learning Sciences/ Education field. However the underlying ideas are broad enough to be useful for researchers in other fields using qualitative research methods.
Surveys and Questionnaires
While remote research can be easily conducted using surveys and questionnaires, some things to keep in mind are: stating the purpose of the survey and number of questions upfront, along with estimated completion time, minimizing the number of questions or getting rid of redundant questions, keeping the questions short and clear, providing visuals along with questions if it helps explain the question more quickly, organizing questions in sections (one section at a time is displayed, then the NEXT button).
Moderated interviews
Moderated interviews work best for smaller number of participants. If more participants need to be interviewed, consider spreading out the data collection over multiple days. As a researcher and interviewer, one could likely experience burnout from interviewing participants more than a few hours each day. If you have a research team or an observer during the interview session, it helps to touch base after each interview and discuss practices or ideas that are working or not working in favor of the research. If you are a solo researcher, it helps to see recordings of the interviews each day and generate 'first-impression' findings or key observations, that could help make the future interviews more rich with the type of data you are looking for.
Unmoderated video response
Your study could have participants directly share their video responses to a prompt provided by you, such as with FlipGrid. This allows participants to do re-takes of their video responses and submit when they feel ready. This ability to edit can also allow participants to think about their responses more deeply, and say the most relevant ideas first. Responses can also be made time-limited, such as asking the participants to respond within two-three minutes.
Moderated interactive workshop
Interactive and moderated workshops are gaining more popularity with activities such as word clouds, digital card sorting, responding to quizzes and showing results in real-time, live editing slides as a group, screen-sharing and interacting with others while playing a game, navigating through a course or learning a new a skill by following-along the moderator, listening to a story or watching a media together with breaks and prompts for discussion, etc.
Observational recordings
Researchers could record observations of the participants and their activities through multiple methods remotely, such as by pre-installing cameras in a research location, recording observational data through digital devices like phones or tablets, or directly asking participants to record themselves while sharing their process of how they design an artifact or perform an activity relevant to the research.
Messaging systems
Remotely accessing group messaging systems such as Slack or MS Teams can allow researchers the opportunities to monitor, record, and moderate interactions among participants. It also helps to perform conversational analysis later because researchers can see the sequence of replies, frequency of conversations, amount of moderation needed, or any other factors that may be of relevance to the research goals.
Forums and Discussion boards
Discussion boards within organizations (like universities, businesses, non-profits) or sub-cultures (such as forums for gamers, artists, musicians, etc.) are a great platform to perform remote research. Make sure to familiarize yourself with the lingo and etiquette of the forums or discussion boards. Make sure to inform new members of the forum how and when conversations will be recorded.
Social media
Facebook groups, Twitter tags, Instagram collections, Linkedin profiles, etc. can all be utilized as data sources as part of remote research study - with ethics and privacy considerations. For example, I wanted to study how parents interact with children's media brands such as PBS KIDS via Twitter - and I collected PBS KIDS related tags on twitter using R and Google Sheets. Basically, every time someone tags PBS KIDS media in a Twitter post, public information such as username of the poster, likes and shares of the tweet, attached images etc. gets recorded in my database. After a few years of data collection, I can potentially look at trends or quality of feedback from parents, responses from producers, any popular characters or shows that get more mention, and so on.
If you are interested in Twitter-based data collection using R, I highly recommend following the work of my friend Bret Willet who sometimes teaches a wonderful workshop on this topic at the Association for Educational Communications and Technology conference. AECT 2021 conference will be held in Chicago (and virtually) in November, come check it out for some great examples of how remote research can be conducted with help of social media!
Recorded media
Recorded media such as film, television, streaming videos, and podcasts can become a powerful data source for some remote research studies, where aspects such as content, storyline, character portrayal, comparison of media, etc. are relevant to the research goals. While content analysis of the media alone may not be sufficient to answer research questions, these could act as an excellent secondary source of data. For example, while preparing a television series pitch for PBS KIDS, I did a foundational research of their content by categorizing series by themes or topics, generated keywords for each series based on TV Guide synopses, creating a visual library of the key characters and environments from each series, and then looked for gaps/prospective new themes and storylines that will fit in with the existing work and overall vision of the organization.
Archives
During the pandemic, several museums made their archives available digitally to the public. Conducting remote research based on archives can be a nice secondary source of data. For example, during my summer research scholarship with the STRONG Museum of Play in Rochester, NY - I studied the archives of construction toys from the last 100 years and studied how the materials, design, tinkerability and supporting materials (such as idea books, instructions, video tutorials etc.) evolved over the years. It was fascinating to see notes and sketches by the designers, accompanying many of the designs. My data included photographs of the toys, blueprints of their product design, notes and sketches by the designers, and in some cases, annual sales data (Potentially to be published next year by International Journal of Play and TASP 2022 conference proceedings).
Diary
Mobile diary studies are popular in UX research, but can also be utilized by other fields. For example, asking participants to log their meals and workouts in a fitness app, logging moods and emotions throughout the day, logging time spent watching television, etc. For the most part participants will work with prompts provided by you as a researcher and the logging of information is usually unmoderated. You could consider sending reminders or follow-up during the journaling/recording/logging process. Remember to specify the tasks for participants in a clear and simple way, and establish how you will record the responses - will the app directly collect and share data with you, or you would like participants to share data manually by self-reporting. You could also consider providing multiple prompts or reminders to make sure the responses stay on track.
Maps
Maps can be utilized in multiple ways to support remote research - such as asking participants to generate mind maps, empathy maps, or journey maps. You could also consider asking participants to map out their activities or interests in relation to any space. For example, my colleague Yuchen Chiu and I developed Learning Environment Visual Mapping, where we try to understand what kind of learning takes place for children and in what kind of environments (settings such as school, home, neighborhood, museums, etc.). You might also consider maps in the sense of the traditional geographical maps, and track participants' movements over a live map, how they could draw shapes on the map by driving around, how they interact with their community through travel, vacation trends, job-related relocation patterns etc.
Artifacts
Collecting actual artifacts or photos and/or videos of artifacts could also be a great way to collect (mostly secondary) data for remote research. Previously discussed maps could also be a nice artifact to collect, to check participants' understanding of an idea, or to understand how they organize or represent information relevant to your research. You can also consider collecting artifacts and ask participants to record a 'making of' video, for a more comprehensive understanding of their process. One of my favorite researchers, Yasmin Kafai often collects research data that involve artifacts (although not remotely) such as checking students' understanding of digital artifacts by looking at the coding behind it (such as MIT's Scratch), or the process of putting together a paper circuit or e-textile bookmark. Recording the artifact itself and its creation process, can potentially support a nice remote study design.
Ethics for Remote Research
Many organizations have Institutional Review Board (IRB) that reviews and then approves, or denies research studies based on legal and ethical considerations. For me, personally, IRB process is a very exciting part of the study because it helps me think more deeply about my research - why is it important, what exactly will I do to collect data, who are some prominent researchers in this area, what will I do if the participants are not comfortable, how long will the study last, will this study benefit any specific participants or a community, how will I recruit participants and will I offer them any rewards, how will I record consent, and so on. The IRB forms are a nice way to organize these thoughts and gives me the peace of mind knowing that I have ran through multiple scenarios, contacted all the relevant people who will collaborate with me, prepared my research questions, interview questions, partnerships with study sites etc.
Ideas for Recruiting Research Participants for Remote Studies
It always helps to establish a relationship with prospective research participants first, be kind to them, try to help them through your expertise, and then it becomes easier to carry on the research as the collaboration becomes more natural and mutually beneficial.
For example, before conducting research at a summer camp, I volunteered to clean-up the arts and crafts area for a few weeks and informally chatted with parents during pick-up/drop off, got to know the teacher facilitators, and developed familiarity with other ongoings of the camp. Eventually when I conducted a research study, all participants and facilitators were comfortable having me around. In terms of a remote study, especially for small groups where you will be interviewing participants, consider helping out with items that can be done remotely. For example, if your research involves interviewing K12 teachers who want to transition into Instructional Designer roles, you could help them by connecting them with prospective ID mentors or share your own knowledge if you have experience in this area. If you are conducting research in a low socioeconomic group, you might consider offering helpful rewards for research participation. For example, if you are interviewing K12 teachers, you might want to consider offering them Target/Amazon gift cards for buying school supplies.
For recruiting larger population of participants, you can consider using tools like MTurk, Ethnic, and L&E (More about tools in the next section!)
Recommended Tools for Remote Study Design
Gathered from this awesome LinkedIn learning course by Amanda Stockwell, checkout these tools:
Video conferencing Zoom, GoToMeeting, Google Meet
Card Sorting Optimal sort
Word Clouds Wordart, WordClouds, MonkeyLearn
Impression tests - Usability Hub
Diary Studies - dscout
Surveys - SurveyMonkey, Google forms. Consider 're-skinning' Google forms to make them more apt for the audience. For example, I modified Google forms to look like a Game for 4th grade math classroom, learn more here
Live recording and annotation - Tetra Insights, Lookback.io
Messaging - Slack, Google Chats
Analytic tools - hotjar, UXcam
Booking and calendar tools - Calendly, Doodle
Integrated platforms for UX research - userZoom, userTesting
Recruiting tools - MTurk, Ethnio, User Interviews, L&E
Running the Remote Research Session
Always remember to test with help of a pilot study to make sure all issues related to the technology, connectivity, flow of questions, understanding of activities, any other research procedures are all working fine. Get consent for recording the session, from all participants in writing prior to the study, and also remind them before starting the study that their responses will be recorded. Remember to stop recording or edit out any personal information that comes in as a surprise, data that was not covered by the IRB and/or not preferred by the participants to be included in the study.
For moderated studies, set a friendly tone and remain neutral so that you don't encourage or discourage participants in sharing their honest opinion. Remember to ask relevant and open-ended questions and allow participants enough time to think and respond. Read facial cues from participants, and pivot questions when needed. Have prepared questions and prompts, but be open to digging deeper into some questions when the opportunity arises!
For unmoderated sessions, consider sending reminders manually or through the app/website/medium of recording itself. During pilot test, make sure that the tasks are understood clearly by the participants and edit if more clarification is needed.
Remote Studies with Children
Conducting research with children can be more challenging compared to adults - getting consent from parents and a verbal assent from children, keeping children engaged and on point, and recording observations or conversations when children are more active (walking/running around). Some ideas to keep children engaged in remote studies is to introduce the research study to parents first, and offering them some tangential activities offline before the study takes place. This could help generate interest in the theme of the research and an emotional investment in the study. While running the remote study with children, also try to include a variety of engaging visuals such as illustrations, animations, storyboards, talking avatars, AR interaction, and any co-creation or guided activities with adults they trust.
If you are interested in learning more about how remote studies with children can be conducted using live talking animated avatars, checkout this tutorial by Adobe, or attend my workshop on animating live avatars at AECT conference in November in Chicago (and virtual). I will be sharing a demonstration of live animation, and providing workshop files so that participants can test out their own projects.
Further Studies
Checkout this awesome course by Amanda Stockwell on conducting remote UX research, and this course by Dave Briss on creating remote workshops