From Observations to Insights: Principles of Qualitative Data Collection and Analysis

Introduction

I am posting this to complement a research panel I was a part of today, which focused on analyzing qualitative data and producing actionable insights.  I wanted a persistent resource that people interested in qualitative research could turn to and use to learn more. I link to a few resources throughout this post.

I break this down into five parts: 1) Five qualities of great qualitative researchers; 2) Principles and metaphors guiding data analysis; 3) Techniques and frameworks for data analysis; 4) Deliverables; 5) Presenting findings. Each of these could be a book on their own, but I try to summarize these with metaphors and examples. Some of the books I list here are great resources for further information. I am also happy to point people in specific directions for further reading, so don't hesitate to reach out with questions as well! 

5 qualities of great researchers

1.     Be Ethical.

Working with people is a privilege, and we should treat their information ethically and responsibly. This means gaining informed consent, protecting their data, minimizing harm, and promoting well-being. These goals are not only in the best interest of your users, but also the long-term success of your company or client. These must be considered from the beginning stages of planning a project, and sometimes this means setting up boundaries for clients regarding access to confidential information gathered from participants, especially in situations where you do in-house, organizational research.

2.     Be Reflective.

 Writing does not come after analysis, it is analysis, and it is a process of reflectionWrite about your emerging insights using memos and jottings. Writing out your thoughts and how the patterns and themes connect is the most concrete way to make sense of your data. Writing is the process of creating order out of chaos, and qualitative data can feel both overwhelming and chaotic early on. Writing about what you see as you analyze your data will help you gain insights from it. This includes being honest and critical of your data, who your audience is, and how it connects back with your objectives.

3.     Be Flexible.

This means go back to the data, use it to guide continued data collection, and be willing to ‘let go’ of your initial ideas and let the data, coupled with theoretical knowledge, guide you. Sometimes new, interesting insights come up that can be used to probe further about unique design opportunities and challenges that weren’t initially considered. Being flexible and reflective means that data analysis is never complete or perfect the first round. It usually requires several rounds of iterative analysis to get it right, and sometimes significantly more, depending on the project, timeline, and budget.

4.     Be Creative. 

This means that you must look for patterns and understand people. Having some sociological, psychological, or anthropological background knowledge can be a huge help in uncovering meaningful insights. Being empathetic and artistic can also aid in this process. Finding the right process for collecting and analyzing qualitative data requires both rigor and creativity – it is both science and art.

5.     Be Empathetic. 

Being empathetic means letting go of your own beliefs, at least temporarily, to better understand those of others; it means being curious and open; it means being ethical and respecting the people you work with and study; and it means reflecting on your own empathy to balance it out with critical thinking. I believe that having diverse experiences and backgrounds are particularly beneficial to qualitative research. One of the best ways to promote empathy is through lived experience, and some qualitative researchers will be better able to empathize with certain people than others as a result. Furthermore, just as anthropologists must navigate and avoid "going native" (a term I don't care for), so too must researchers keep an eye out for their own empathy.

Principles and metaphors guiding data analysis:

  • Treat your data like a romantic partner.

I was socialized to treat my data like a significant other – get to know it, spend time with it, talk to others about it, and respect it. Immerse yourself in the data and the lives of those whose data you work with, and care for it in ways that are ethical: there are real people behind it. Also, don’t smother it; give it some space, take some time away to clear your mind. Finding the right balance is crucial, and spending time away from your data can give you new insights and appreciation for it.

  • Data analysis begins with data collection.

It is often best to begin data analysis at the same time as data collection, or before, when you are developing your research questions and design. When possible, this means debriefing with designers between interviews or at the end of a day traveling to a new location to discuss what you found insightful. In some cases you want to wait until you are done collecting data to analyze it, but it is often best, especially when doing field work, to work with your data as you collect it. This helps guide your data collection moving forward as contexts and practices become clearer.

  • Work with data while it’s still fresh.

The longer you wait to begin writing up notes, entering interviews into the team’s work flow, and reviewing and making memos about your field notes, the more details are forgotten. This detail decay begins IMMEDIATELY after the session and accelerates over night. DO NOT WAIT.

  • Break data down into bits.

Interviews should be broken down into data points that enable each bit to be moved, clustered, and reassembled in new ways. This can be done through a process of structural coding, where you go line-by-line and mark data by theme or research question. Themes could relate to motivations, behaviors, emotions, attitudes, goals, values, mental models, relationships, In Vivo codes (quotes and slang used by participants), etc. I give some examples of these below.

  • Data must be organized and traceable.

Each data point, such as a quote, observation, or artifact (e.g. photos) needs to be organized and ascribed a session code, location, method of data collection, participant, and researcher who collected the data. This allows you and others to trace it back to the source in case you need more context or information from that session.

For example: “Both my son and daughter have a phone and laptop, even though they are only 8 and 10 years old, because I want them to be prepared for jobs in technology when they grow up.” Kiara-GA-H.INT-P03-MD (depending on your needs and organization, you can vary in detail. Here, we have a pseudonym for the participant, who is from Georgia, data was from a home interview, 3rd participant interviewed by Michael Dickard – this is searchable in our Google Drive, where all data is stored.)

  • Touch your data.

I prefer moving data off of my computer and printing it on note-cards so I can see patterns more clearly and map it out spatially. While working at Yahoo, we used this Word Macro to aid in this process (must have developer mode enabled in Word to use it): http://cs377u.stanford.edu/Normal.dotm

  • Work with others.

This isn’t always possible, but it is often best to work on analysis with at least one other person, and to get feedback from others on the team. This could be as simple as showing them your clustering of notes to see if they make sense. We are all biased, and we each have blind spots. Getting feedback and doing reality checks through iterative data collection and by working through your insights with someone else will pay off. It helps to do affinity mapping and move data around on walls with others present as well – but consider the pros and cons of this: some people are more outgoing and can quickly take over an analysis session. Consider quiet time to work with data, then come together as a team to discuss your themes and categories.

Techniques for analyzing qualitative data

Above I gave some guiding principles and metaphors for conducting qualitative data analysis. Here, I will focus on some more concrete techniques. Which ones you use will depend on your research questions, the constraints of your project timeline and budget, and your own skills and preferences. Your goal when analyzing data is to provide depth and context to team members, find and confirm patterns within the data, interpret these patterns, and create analyses that inform action.

Academics and UX researchers often use thematic analysis, grounded theory, and affinity mapping to analyze their data. Each of these have rich bodies of literature supporting them, so I won't dive into them here, but recommend searching for further reading on these. Instead, I will focus on a few categories that often produce useful insights. Consider looking for some of the following when analyzing data:

  • Mental models: These are how people make sense of and represent the world. Understanding how people think about a piece of software or how they think systems and institutions work are extremely useful information for designers. For example, some people think of tools as people, such as a helper or friend, or give them personalities, such as a silly robot. In other cases, people use metaphors to guide their behaviors - listening for how they frame things, and the language and analogies they use, are often gold mines for insights.
  • Values: People's behaviors are often guided by what they value, and it is important to understand these systems of likes, dislikes, beliefs, and associations they make between objects, people, and their own personal situations. Do people prominently display photos of trips with loved ones in their living room? What does the physical wear of a device tell you about a product they use? Which apps do they keep on their home screen for quick access and why? Keeping a keen eye for details such as these will help you better understand what people's values are, and help prioritize and guide how you design for them or brand your product.
  • Motivations and Goals: What functions do behaviors serve? What do people want to get out of a product or activity? Why do they behave the way they do, and what drives them to take action? For example, when feeling lonely or nostalgic, people may be motivated to reach out and reconnect with an old friend. To do so, they send them an old photo in a text message and tell stories as a way to reconnect.
  • Behaviors: What actions do people perform? Are these behaviors regular or irregular? Regular behaviors are often rituals that support various goals and function in unique ways. Understanding such behaviors and rituals can be insightful and provide a deeper understanding to the structure of people's practices. Looking for moments of delight, pain, or regularity, understanding what people do, and what the circumstances are that trigger specific actions, are important starting points for design.
  • Roles: What is someone's role within an organization, team, or relationship? What functions do they serve? You may find that sorting people into roles can help you uncover archetypes and personas that guide system design. These could be mothers, fathers, and children and how their roles guide interaction around digital photo albums in the living room. Maybe the mother is a "storyteller" and the father a "teacher," each drawing on photos for different reasons when interacting with their children.

For brevity's sake, I'll stop here and say that during interviews, you may want to probe people when you notice some of these categories pop up, if it is appropriate and central to your research questions, as these will often guide your data analysis in various ways. For example, when strong attitudes are expressed towards other people, practices, organizations, or situations, people often have a specific event in mind - ask them if they have something in mind to better understand how these stories shape their attitudes and beliefs.

Frameworks

Once you have gone through one or two rounds of initial analysis using some of the categories above, you can begin relating these categories to frameworks and cluster them into themes. Some common ways to present these visually include taxonomies, networks, maps, journeys, timelines, flowcharts, and matrices. These frameworks are not your final product, but rather should be used as props to aid story telling. Pulling together quotes, photos, artefacts, video clips, and one of these frameworks can provide compelling stories to your audience. The primary purpose of these frameworks is to map out actionable recommendations, and stories are meant to give them emotional power.

For example, you might create a matrix with columns that include categories for feelings and concerns, what an archetypal user does most, problems they typically run into, and areas of opportunity. In each row, you can include the 4 archetypal users you identified (you may have a separate matrix describing some other key characteristics of these people). For further reading on how to construct these frameworks and produce actionable insights, do some light searching on these frameworks. Two solid books for reference: Qualitative Data Analysis: A Methods Source Book and Observing the User Experience.

Primary Deliverables from data analysis:

Deliverables range from soft to hard. Soft deliverables are secondary, but still important, outcomes of research, such as building friendships with team members, developing empathy for users, transferring skills to others, and developing myths and stories that guide design within your company. Hard deliverables range from foundational to evaluative reports.

 (Note: Some of these deliverables and quoted content come from the The Field Study Handbook. This is a great resource when coupled with other more in-depth books on research methodologies.) 

Soft Deliverables:

  • Friendships: Bonding with the team, or individual, you are working with, whether through shared experiences traveling and collecting data, or through iterative rounds of data collection and building prototypes to test – this can carry over to future projects and build trust and camaraderie in a team.
  • Empathy: Gaining a more nuanced, empathetic understanding of other people’s lives can make your product better, more accessible, more delightful, and help your company meat its goals as well.
  • Skill Transfer: Often, going through this process of data collection and analysis with others can begin to transfer skills to other members of the team.
  • Myths and stories: Stories are central to who we are as well as guiding our behaviors. As a researcher, it is often our job to dispel pervasive and enduring myths about users, and introduce new myths or stories that teams can rally around and guide product development. Sometimes these myths come from the mission statement, or the foundation of the company; other times they come from research, portraying people in an empathetic light and providing anchors to build narratives around.

Hard Deliverables:

  • Foundational Reports: “Why is this interesting? What, how, and why are people living the way they do?” – this type of report is about depth and context, often in a new domain, that builds an understanding of what people do, how, and why. This provides perspectives that are not always obvious to outsiders and even overlooked by insiders – in Sociology, I learned to take note of the everyday, mundane and make it interesting. What people think of as trivial or common sense can provide deep insights to your project. When done well, foundational projects can be packaged into personas and archetypes, which often serve as the basis for future questions and indices in segmentation surveys.
  • UX Report: “What people do, how and why they do it. How it can be done better.”
  • Usability report: “What are people’s mental models? Is the design usable? Is it useful to them right now?”
  • Evaluative Report: “These are the issues people have, and how they can be addressed."
  • Strategy: “These are the business opportunities and risks, and how we can address them."
  • Brand report: “This is what we stand for, these are our values.”

Communicating and Delivering Results:

When presenting deliverables, I mentioned the importance of using frameworks, personas, archetypes, and stories to build empathy and guide actionable outcomes. How you frame these stories is important. Instead of creating an archetype that says "This group is highly driven but unorganized," write: "This group of people are potential power users, but need help organizing and paying their bills on time." Don't just tell, but show your stakeholders why these people are important, who they are, and how these insights are actionable. Other ways to do this are using scenarios and triangulating findings with multiple studies (e.g. in-depth qualitative interviews and field work followed by large-scale surveys that produce segments and identify broader prevalence and patterns.)

It is also important to build trust over time, if you are an in-house researcher, by working closely with designers, product managers, and engineers. I would highly recommend keeping them in the loop, learning how to frame your findings for different audiences, and using your research skills to understand the dynamics within your organization. All of these tips apply to doing research to understand users as well as your own team. Learn the goals and motivations of others so you can help diverse people out with your research. One size doesn't fit all here, and this requires some experience and practice. I'll end by suggesting this article by some Facebook researchers on how to communicate findings to your team.

This ended up being a little longer than I planned, and this is just the tip of the iceberg. I will follow up with some posts in the future to do some deeper dives on different aspects of data analysis and data collection.

Hope this was helpful!