When Should You Call a Professional?
by Minda Borun
The Franklin Institute and Museum Solutions
The Evaluation Sequence
Evaluation studies involve a sequence of tasks. There is study design, data collection, data entry, tabulation, analysis, and report-writing. The most important jobs for an evaluation professional are study design and data analysis. Study design should be a collaborative process involving the evaluator and museum staff. It will set the course for the ensuing activities and determine the worth of the eventual report. Appropriate design of the assessment requires that the research questions and techniques match the goals of the project and answer the relevant questions. Once the study and data collection instruments are developed, staff can be trained to collect, enter, and tabulate the data. The professional should return to analyze or make sense of the data, to determine what it shows. Then staff or professional or both can write up the report. Each of the six tasks listed above apply to each of the four phases of evaluation: front-end, formative, remedial, and summative.
Front-end evaluation, occurs in the early planning stages of a project and usually involves focus groups and/or surveys to determine what your visitors will bring to the experience--what they know about the topic, what misconceptions they may have, what points of view or themes are most interesting, what questions they want answered, and how they feel about specific ideas you have for exhibit or program experiences. It’s important to have a professional design or edit survey questions, develop the topic guide for the focus groups, and moderate the groups. Professional moderators will know how to elicit participation from all members of the group, keep one person from dominating talk time, and keep the discussion on track.
Formative evaluation involves successive testing and modification of prototypes or mock-ups of exhibit or program experiences. The mock-up should be a rough version of the final experience and have a similar impact on the visitor. Initially you are testing for failure, so small samples suffice. Once it’s clear the something doesn’t work, it is modified and retested. It’s only when the experience or device seems to be working well that a larger sample of visitors is desirable to make sure that it works for the desired range of visitors. Again, it’s important to consult an evaluation professional in designing the prototypes and the testing instruments and to review the results of testing.
Remedial evaluation is formative evaluation of the installed exhibition. You are still looking for things that don’t work and have reserved funds to fix them. At this point, you can look at visitors’ use of thematic groupings of exhibits, determine whether or not the main themes or areas are seen and understood, observe whether traffic patterns take visitors to all parts of the exhibition or if there are significant areas that are being missed, and find out visitor’s response to the lighting and ambience of the exhibition.
Finally, you need to conduct a summative evaluation to assess the impact of the completed exhibit. This usually involves a tracking and timing study to see where visitors go and how much time they spend at the various exhibit components. There is also an exit interview to determine visitors’ reactions to the exhibit and the meaning they make of it. As above, a professional evaluator is needed to teach staff to do the tracking and timing study and to help to interpret the results. You also need a professional to design the exit interview and to analyze the tabulated data in order to accurately interpret the results.
Exhibit and program evaluation is best thought of as a collaborative endeavor involving professional input, staff or volunteer labor and professional analysis. It’s important to distinguish the various phases of a study and not to confuse data collection with overall study design.