Small data, big impact: Making the case for qualitative research in libraries

Surveys are undoubtedly a popular approach for collecting data from service users. And while they can seem like an easy approach, designing a good questionnaire can be harder than it looks. You are not just selecting what you want to ask, but thinking about the right question format, sample sizes and how you’ll interpret the results. 

But even a well-designed survey has its limits. Most surveys lean heavily on quantitative data for their ease of analysis, but this can be deceptive. As design researcher Erika Hall warns, "Surveys are the most dangerous research tool — misunderstood and misused." The ease of creating and distributing them, and tallying up the results, can make findings feel true and valid even when they're misleading.

There's also the burden on your users to consider. Survey fatigue is real. Because surveys are relatively easy to put together, they're often a first choice. And they often end up being much longer than they need to be. But your users are already busy, and another form to fill in might not be especially welcome.

So what's the alternative? Qualitative data is often overlooked, which is a shame, because it gives you much deeper insight into users and their experiences. It lets people describe things in their own words, rather than choosing from options you've already decided for them. That's why when it comes to a survey we're advocates for a single, well-chosen open-ended question.

Satisfaction and impact are not the same thing

Before you can ask the right question, it helps to be clear about what you're actually trying to measure. Most library evaluation sits in one of four categories: activity (what happened and when), attendance (who came and how many), satisfaction (did people enjoy it or find it useful), and behavioural change (what people actually did differently as a result). Some data can be relatively easy to capture with closed questions or by ticking the boxes. But capturing impact and measuring the outcomes requires a different approach.

Sometimes we confuse satisfaction with impact, but they're not the same thing. Knowing that high numbers of attendees rated a workshop ‘good’ or ‘excellent’ might mean that the majority of them were happy with how you organised and ran the workshop. But It doesn't tell you whether they applied anything they learnt, changed how they work or felt more confident as a result.

When measuring impact, asking a single open-ended question is often the most direct and effective way of letting your users tell you about the difference your library’s services have made in their lives.

So, what question do we actually ask? In order to get the most meaningful answers that measure outcomes, we need to focus our attention on a user. Instead of asking “how would you rate today's session?”, you can try “what will you do differently as a result of today?” or “what's one thing you learnt that you'll actually use?” Questions like these ask people to reflect on their own experience, and this is when you usually get meaningful responses. 

One of the benefits of asking open-ended questions is the discovery of things you didn’t know to ask about. Another benefit is that they don’t lead respondents towards answers you've already assumed. The responses you collect are often more authentic, which may help bring  your stakeholders on board.

There is sometimes a misconception that qualitative data can’t be measured and is therefore less valuable than quantitative data. But collecting responses via a single open-ended question doesn’t mean you can’t demonstrate impact in a meaningful way. You can turn individual stories into valuable insights by identifying patterns and common themes. Qualitative methods such as content analysis and thematic analysis make it possible to systematically analyse qualitative data, identify trends, allowing you to draw clear and evidence-based conclusions.

A single, well-crafted open-ended question won't give you a dashboard of statistics. But it will give you something more valuable: evidence of change, in people's own words.