Family Learning Forum

A Project of the USS Constitution Museum

How Much Data Do You Need?

by USS Constitution Museum Team

Introduction

So you’ve decided you want to conduct a visitor survey. Now what?

It all depends on what kind of information you want to collect.

Quantitative data is largely objective—things that have definite, measurable answers. It includes:

  • Visitor demographics (age, race, zip code of residence, etc.)
  • How much time a visitor spends in a given gallery, or at a particular label or interactive
  • “Yes” or “No” questions (“Have you ever been to this museum before?”)
  • Multiple choice questions, or questions with a set number of potential responses (“How did this presentation make you feel? Circle one [or all answers that apply].”)

Qualitative data is more open-ended. It includes:

  • Why a visitor prefers one exhibit, label, or activity over another
  • What a visitor would change about an exhibit, label, or activity
  • Knowledge gained from an exhibit or program (“Name two things that you learned about [subject] from this exhibit/program.”)

While this is by no means an exhaustive list, if you’re new to the world of surveys and visitor analysis, hopefully you’re beginning to grasp the differences between quantitative and qualitative data. The general rule is that quantitative data requires more responses than qualitative data in order to be considered reliable and useful.

Now…what kinds of data will you need when conducting each of the three types of evaluation?

Front-end evaluation can be both quantitative and qualitative. You might be looking to answer basic questions about the demographics of your visitorship—their ages, whether they are tourists or locals, how many times per year they visit your institution, etc. Qualitative is obviously useful for this type of information. On the other hand, you might also want to find out what they’re interested in, and what kind of programs they’d like to see offered in the future. “What topic would you like to see [Museum] explore in an exhibit?” (as an open-ended question, with no possible choices) is an example of a qualitative front-end question.

Formative evaluation is largely qualitative, but can also include quantitative elements. Since a large part of this stage often involves prototyping, you’ll want to ask your visitors what they prefer, what they like or dislike, and most importantly, WHY they feel this way and HOW they would change things. For example, you might be testing the effectiveness of an interactive: it’s not enough merely to ask, “Do you like this activity?” You’re going to want to know more detailed information: “What do you like (or dislike) about it? What, if anything would you change?” Often, formative evaluation is used to test temporary signage and other aspects of an exhibit or program as it gets started.

Summative evaluation is, like front-end evaluation, both qualitative and quantitative. At the end of an exhibit/program design process, this data tells you how visitor behavior, knowledge, and interactions have changed after participating. Summative evaluation includes timing and tracking studies and post-visit focus groups or phone calls. Certain information (for instance, how long a visitor spends at a particular interactive) can be recorded objectively by a staff member or volunteer with a stopwatch, pencil, and a floorplan of the exhibit. However, in observing visitors, it’s also crucial to record notable quotes from your visitors, as well as how they interact and behave in the exhibit and with each other.

Evaluation

Basic Statistical and Evaluation Terminology

If you read an evaluation report from a museum or research organization, you may run into a lot of terminology that you’re unfamiliar with. This short glossary (from “Evaluation as Investment,” by Dr. Elee Wood and Dr. Barbara Wolf) should be an introduction to some basic statistical terms.

  • Sample: a portion of the whole group being studied. Sampling, then is a process you use to get a sample. Random samples are use a predetermined systematic process.
  • Population: the total of the group being studied. A population can refer to all of the people in a particular category, e.g. all visitors to the museum exhibit; all visitors to a museum; program participants.
  • N or n: The number of participants or respondents.
  • Variable: any factor that has an impact on a given situation. There are dependent (the ‘x’ variable) and independent variables (the ‘y’ variables). Dependent variables are the ones we study. Independent variables are the factors that impact or influence the dependent variable.
  • Significance: in order to make meaning and define relationships, we can determine how different or how similar values are. Significance is the degree to which something is or isn’t meaningful. Something that is “statistically significant” has particular meaning and assumes that the correct procedures and methods have been used to produce the result.
  • p value: often expressed as p = ; p ≤ The number following the p represents the percentage of chance that a value is related to chance, rather than the variables in the study. For example, a p value of .05 means that there is a 5% chance that the difference in the value is due to chance rather. In general the lower the p value, .01, .001, etc. indicates there is a very low probability that a difference in the value is due to chance.

How Much Data Do You Need? Expert Opinions

From Monica Post, MPR Museum Consulting:

“Determining sample size is controversial.  The larger the sample, the more accurate it will be, but also more costly.  My mentor, Beverly Serrell, believes that by the time you test 20 people you will see the trends.  She usually chooses to test an additional 20 people to verify that the data is accurate.  I tend to follow the same philosophy, but I am more comfortable with testing 40 to 60 people.  It depends on the project.”

From Scottish National Heritage:

How many people do I need to interview or observe?

Some guidelines on your sample size are as follows:

  • Evaluation is always a trade-off between time, money and accuracy. The bigger the sample size, the more accurate the results are, but the more it costs.
  • For formative evaluation quite small numbers are all that’s required – if something doesn't work it will show up quickly.
  • Qualitative research usually involves smaller sample sizes, for example in-depth interviews of 20 visitors to a site.
  • Quantitative research involves bigger sample sizes, such as interviews of 150+ visitors to a site. To ensure the sample is statistically representative interviewees must be chosen at random. When interviewing groups of people the ‘next birthday’ rule helps here – find out whose birthday is next and then interview them, rather than the dominant ‘group leader’.
  • Focus groups typically involve 8 -10 participants from one particular visitor group, such as social class ABCI or urban fringe dwellers. Several focus groups may be needed to reflect the visitor profile of a site.
From Nicky Boyd, Museum Learning and Education Consultant:

Qualitative data

  • Involves smaller sample sizes. You will find that you don’t need to consult very many people before you start getting the same kind of information
  • A smaller sample is more appropriate when the purpose is to gather ideas or identify problems and issues  
  • You don’t necessarily need to ensure you consult a wide range of people

Quantitative data  

  • Involves larger  sample sizes 
  • You need to be ask at least 100 people before you can start expressing results as percentages
  • A rule of thumb is to aim for 10-20% of your total group
  • Consulting a greater number of people does not make your results more valid or more representative. What is more important is who you ask – you need to ensure they are representative of the group or audience overall

From Nick Visscher, Internship Coordinator and New Directions in Audience Research Coordinator at the University of Washington’s Museology Program:

“In my opinion the amount of data you collect is dependent on why you're collecting it and how you're going to be using it. If you're doing prototyping or other formative study a smaller amount of visitors would be required because you're looking to make changes, maybe multiple changes throughout the process...for something more formal and summative (and something you might be interested in generalizing) a larger and more representative sample would be required.

Typically, in our experience if a study includes around 100 participants... it can be considered a strong and solid evaluation.  We like to use the chart offered in the book "Practical Evaluation Guide: Tools for Museums and Other Informal Educational Settings," by Judy Diamond, Jessica Luke and David Uttal.
Essentially if you're willing to accept -/+ 3% sampling error, which is probably more strict than most evaluation studies, the sample should be around 1000 participants [regardless of your total population].   For a 10 percent sampling error, [which is] more common, once you hit a population of about 5,000 the number of people you need in your sample remains constant at about 100.”

From Jeff Hayward, PhD, Director of People, Places & Design Research:

Many people will probably tell you that for formative evaluation studies, sample sizes of 35 to 50 visitors is usually sufficient. 

However, it's also important to understand that the composition of the sample is important -- if it's way off, for some reason (e.g., the sample was members, but the project wants to generalize to general public visitors; or the sample was school group chaperones), then the results could be misleading.  All visitors are not the same.