Effective questionnaires for all
A step by step recipe
for successful questionnaire
Step
1 – First Define Your Project
As
with all evaluation before you do anything else answer these questions in this order:
1. What
do I want to find out? Who do you want to find this out from?
2. Why
do I want to find this out?
3. How
will this information be used?
4. How
will I find this out – choose your methodologies (note the plural) and sample?
5. Who
am I finding this out for and how shall I tell them the results?
6. How
much money do I have for this project? How many staff? How much time?
Step
2 – What Type of Survey Do You Want?
OK
so you have chosen to do a survey. Here are your options for the type of survey
you might want to conduct:
Self-completion questionnaires
pick-up
– left lying around on tables or in rubbish bins
on-line
– via email or pop-up windows on web pages
Interviews
face-to-face
telephone
postal
email
And
here are some hints to help you make your choice of survey
Self-completion questions
Pros Cons
quick no control over sample
cheap exclude people who can’t write mostly
get negative comments
often
very low return rate
Interviews
Pros Cons
can
get representative sample time consuming
can
probe for more detailed / useful answers costly
in terms of staff and money
Summary
Self-completion
questionnaires are cheap but largely useless. You really don’t have any control
over your sample.
With
self-completion questionnaires you have no chance of obtaining a representative
cross-section of your audience. Children, hassled parents with children, people
without a pen, people who cannot read or write English and people who are
generally happy with the Museum, will not be included in a self-completion
survey. Instead you will end up surveying a few highly motivated individuals
who are often incredibly pissed-off about something. Interviews are more costly
and time-consuming but give you much more reliable data.
The
only time it is worth using self-completion questionnaires are:
·
When you want to do a quick pilot study
·
When you want to develop a formal
customer complaints system
·
When you can ‘force’ a representative
sample to complete the questionnaire e.g. a school group who will be
‘encouraged’ and supervised by a teacher
A word or two about postal, email
and web questionnaire
Postal questionnaires
Postal
questionnaires can be a very effective method of interviewing people but you must
get a return rate of at least 60% for the data to be consider reliable.
Your
first letter will elicit a return rate of about 10-20%. Your second (reminder
letter) will increase this to about 30-40%. Hopefully your third reminder (plus
repeat copy of the questionnaire for all those people with hungry dogs) should
get you a return rate of 60% plus.
Adding
an incentive helps – but not all that much and be careful there are legal implications.
Prize draws are now tightly regulated.
Email questionnaires
We
have found email to be a particularly effective method of surveying adult
visitors (NB around 80% of the Science
Museum visitors have access to email at home or work. However, be warned this
is not the case for many segments of the population).
Email
questionnaires we use are entirely within the body of the email i.e. they are
not an attachment.
We
have found that Friday’s are the best days to send an email questionnaire since
many people receive and answer them at work. We generally get a return rate of
50-75%.
Web surveys
Web
surveys – where a questionnaire pops up when someone enters a web-site or a particular
part of a web-site –generally get a low response rate. On average only 2-3% of
people visiting a web-site will complete such a questionnaire. A 5% return rate
is considered to be very good. This means that data from such questionnaires
must be treated with caution as it is likely that segments of the audience are
not being included in the survey for one reason or another. To be fair though
pop-up surveys are sometimes the only way to collect data about a web-site’s
audience profile.
It
is worth adding a ‘cookie’ to the questionnaire so as not to annoy regular
visitors who would otherwise be sampled every time that they enter the site. The
use of a cookie should also prevent people from giving multiple replies but
this system is not fool-proof and many people set up systems to stop cookies
being loaded onto their machines.
Step
3 - Quantitative or Qualitative?
Do
you want data that you can analyse statistically or do you want very detailed
in-depth data? You need to decide before you choose your methodology since this
will determine the type of questionnaire you design and the size of sample you
will take.
Here
are some questions that will help you decide:
1. Do
you want to know how many people think or do something?
2. Do
you want to find out what the majority view is?
3. Do
you want to find out how views/behaviour vary according to age, gender, frequency
of visiting the museum?
4. Do
you need statistical data to convince your audience?
5. Do
you want to understand why people do or think something?
6. Do
you want to find out in detail what people think about something?
7. Do
you want to find out about people’s emotional response to something?
8. Do
you want to explore the range of views that your audience have?
9. Are
you expecting to collect a lot of data from each respondent?
If
the answer is yes to questions 1-4 you need quantitative data. This means
interviewing a sample of at least 100 people. But remember size of sample does
not necessarily equate with quality of sample. The sample needs to be an
accurate cross section of your audience and not exclude people who only visit
on weekends or who don’t enter the building by a particular entrance.
If
the answer is yes to questions 5-9 you need qualitative data. This means
interviewing people in much greater depth.
The sample still needs to be carefully and rigorously selected but it
will be smaller.
Examples
Quantitative
research would be required if you were trying to find out …
·
Average length of stay in museum
·
Whether the type of gallery visited is
affected by the age of the visitor
·
Whether visitors with experience of
using the internet have a different perception of the exhibition
Qualitative
research would be required if you were trying to find out …
·
Why visitors of different ages want to
visitors different types of gallery
·
The range of views visitors hold about
an exhibition
·
Why visitors have such difficult
understanding this exhibit
Small does not mean sloppy
Just
because a qualitative sample is small does NOT mean that is should be sloppy. You
should be just as rigorous in collecting a small sample for qualitative
research as a large quantitative sample. Qualitative research is no less rigorous
than quantitative research.
Summary
The
choice of quantitative or qualitative data should be driven by the needs of the
project. It may well be that you need both types of data in which case you have
to design more than one type of questionnaire and sampling strategy.
Basically
quantitative research is about finding out about what is happening while qualitative data is about finding out why something is happening.
Step
4 - Choosing Your Questions (The Big Step)
Think
about your visitors! What are they going to make of your questions? Will they understand
what you are after? Are they willing and able to provide this information?
Visitors
are not passive recipients of questions who respond in predictable ways to external
stimuli. Visitors are trying to work out who you are, why you are asking these questions,
how they appear to you, how they feel about themselves. They will be trying to
work all this out from how you introduce yourself, your reaction to them, the
questions that you ask and the options that you offer them. There are many
opportunities to mislead and confuse visitors and this can have a dramatic
impact on the quality of your data.
I
would therefore recommend two key rules;
i.
Be clear about what you are trying to do
ii.
Clearly explain what you are trying to
do to the interviewee
Ultimately
the objectives of your research will decide your questions. Everything must be checked
against these objectives. One useful strategy is to add to your first draft of
your questionnaire a description of why you want to ask each question and what
sort of answers you are expecting to get.
However
you must also be sure that your interviewees understand your questions in the way
that you want them to and can give the answers you require. This depends not
only on the wording of your questions but also on your initial introduction,
any further explanation you provide, who you interview, where you interview
them and many other factors.
When
you choose your questions you need to consider many factors. The following section
will give you some broad outlines of what to do and what to avoid but the best course
of action is to carefully test your questionnaire before you launch your full
scale survey (see step 7). What you need to do can be summarised by the acronym
TAP
Topic –
the topic should be clearly defined for you and for the respondent so that you are
both understand what is being talked about
Applicability –
The applicability of the question to each respondent should be established. Respondents
should not be asked to give information that they do not have
Perspective – the
perspective that the respondent should adopt when answering the questions
should be specified
Types of question
There
are two main types of question and you need to be clear which you want to use and
when. Have a look at this scene from everyday life.
Darren
– aged 16 –lives at home with his parents Bob and Shirley. He has just been out
with his girl-friend, Tracy, to see a film and has returned home around 1.00 am
to find his parents are still up. Here’s what happens next …
Bob You’re home late?
Darren Yeah
Shirley Did you have a good time?
Darren Yeah
Bob Did you see that film?
Darren Yeah we did
Shirley Was it good?
Darren Yeah it was OK
Bob Did Tracy like it?
Darren Yeah
Bob So it was worth seeing
then?
Darren Yeah it was worth seeing
Shirley I‘ve heard a lot about it.
Do you think I’d like it?
Darren I don’t know maybe (long
pause) … well … I’m off to bed. Night.
Bob Night son
Shirley Goodnight sweetie-pie
Shirley I don’t know what’s wrong
with Darren these days he hardly seems to have two words to say to us.
What
problems were Bob and Shirley having communicating with Darren? Was it just his
age? Was it his hormones? What was wrong with their questions?
Open or closed?
Open-ended
questions are those where visitors have to answer in their own words for example:
If
you where describing this exhibition to someone who hadn’t seen it yet what would
you say it was all about?
What do you think we
could do to improve this exhibition?
Open-ended
questions:
·
provide very rich data
·
they do not constrain visitors to
answering in your terms
·
elicit much more detailed answers
·
gain greater understanding of
interviewee’s opinions
·
allows interviewee to raise issues you
did not think of
But
…
·
difficult to record answers – may be
better tape recording interviews
·
difficult & time-consuming to
analyse data - need to categorise answers
·
more difficult for visitors to answer
·
interviewees may not give a relevant
answer
·
interviewees may not give a full and
comprehensive answer – you must:
o
clearly define the scope of the question
o
explain the perspective you want the
interviewee to take i.e. your point of view, you and your family; your
community’s points of view etc.
o
probe to ensure that interviewee has
fully expressed their point of view and
Closed questions
only offer (or appear to offer) a limited set of answers for example:
Which of the following
age categories do you fit into?
How much did you enjoy
this exhibition?
Closed
questions also include the following type of question:
Yes/no questions
Have you visited the
Science Museum before yes/no?
Number scales
On
a scale of 1 to 5 where 1 is not interested at all and 5 is very interested, how
interested would you be in using this exhibit in an exhibition? (and don’t worry
I didn’t make this exhibit so I wont be offended by what you say)
1 2 3 4 5
Rating scales
How
would you rate the following facilities provided in the Science Of Sport
exhibition? I would like you to tell me whether you think they were very good,
good, average, poor or very poor.
|
|
Very
Good
|
Good
|
Average
|
Poor
|
Very
Poor
|
Don’t
know
|
|
Staff
|
|
|
|
|
|
|
|
Hands
on exihibits
|
|
|
|
|
|
|
|
Information
panels
|
|
|
|
|
|
|
|
The
shop
|
|
|
|
|
|
|
|
Value
for money
|
|
|
|
|
|
|
Scale of agreement
I
am going to read you five statements I would like you to tell me whether you
strong agree, agree, disagree, strongly disagree or neither agree or disagree
with each statement
|
|
Strongly
agree
|
Agree
|
Neither
agree nor disagree
|
disagree
|
Strongly
disagree
|
|
Statement
1
|
|
|
|
|
|
|
Statement
2
|
|
|
|
|
|
|
Statement
3
|
|
|
|
|
|
|
Statement
4
|
|
|
|
|
|
|
Statement
5
|
|
|
|
|
|