The perfect survey

In this article you will learn to consider the key ingredients that make up an effective survey.

When writing a survey, it can be tempting to dive right in and start throwing questions down. We'd advise a more thoughtful approach to ensure you can draw meaningful conclusions form your survey results. For example, if you ask the wrong questions, you'll struggle to draw conclusions. If you have the right questions but you phrase them poorly, you can get unreliable answers. If you get the questions right but you structure the survey badly, your completion rate can fall dramatically.

Thoughtful survey design is a well understood topic with plenty of resources and guidance available to you. Here, we focus on the key ingredients that will give you confidence in your surveys and your results.

Types of survey question
Survey structure
Effective question phrasing

Types of survey question

The Anywhere streams offers a variety of question types. Let's take a look at each one with a focus on what you might use it for.

Smiley Question

The smiley question is a virtual version of the physical ExpressPod smiley terminals. It uses a five-point scale that ranges from 'very unhappy' to 'very happy', with each answer choice represented by a smiley emoji.

An example of a smiley question is: "How was your experience today?"

Results
While you can see how many people chose each emoji, the real power in a smiley question lies in something called its 'index'. This is a single number that Ombea calculates from the results. The index represents the balanced total of all your results. Ombea offers four index types:

- Insights Index ranges from 0 to 100.
- Net Promoter Score (NPS) ranges from -100 to +100.
- Customer Effort Score (CES) ranges from 0 to 100.
- Customer Satisfaction (CSAT) ranges from 0 to 100.

In all cases, the higher the number, the better the sentiment amongst those people that answered this question. You can learn more about how each index is calculated
here.

Usage
You would use a smiley question when you are asking people to 'rate' something such as an experience, a service, a feeling, or some other kind of interaction.

Multiple-Choice

A multiple-choice question is a question where the respondent chooses from a list of possible answers.

An example of a multiple-choice question is:
Which age category do you belong to?

- Under 18
- 18-24
- 25-34
- 35-44
- 45-54
- 55-64
- 65 and over

In the example above, it's clearly impossible for the respondent to answer more than one.

Results
A multiple-choice question can have any number of answer choices (although keeping it simpler is often better!) The answers are represented in Ombea as a bar graph showing how many people selected each answer.

Multiple-choice questions cannot be turned into an index because the answer choices usually represent related but entirely separate categories.

Usage
You would use a multiple-choice question to group your respondents into different demographics, or to understand a single preference from a list. Multiple-choice questions can be turned into statistics very quickly because there is a fixed set of answer choices.

Multiple-Response

A multiple-response question is an extension of the multiple-choice question. The twist is that this time, the respondent is invited to choose more than one response, should they wish to.

An example of a multiple-response question is:
Which of the following continents have you visited in the last 12 months?
a. Africa
b. Antarctica
c. Asia
d. Europe
e. North America
f. Australia
g. South America

In the example above, it's clearly possible for the respondent to answer more than one.

Results
A multiple-response question shows up in your dashboard as a bar graph showing how many people selected each answer.

Multiple-response questions cannot be turned into an index.

Usage
You would use a multiple-response question to understand preferences or choices from a list where it's possible to have more than one answer. Multiple-response questions can be turned into statistics very quickly because there is a fixed set of answer choices.

Free-Text

A free-text question is a completely open question that invites the respondent to submit any answer they wish. They are presented with a text box to type in whatever they like.

An example of a free-text question is:
Please provide any additional comments that will help us improve

In the example above, the respondent can type anything that comes to their mind. While you could use a multiple-choice question, this might be less valuable because the answers would be limited by the choices you present.

Results
It's not usually practical to present the results from a free-text question as a graph. Instead, Ombea presents the results either as a table of answers or as a word cloud. A word cloud is a special graphic that shows all the words, with more frequent words presented in a larger font.

Usage
You would typically use a free-text question in the following scenarios:
To gather ideas.
In the early stages of collecting research, to understand what people think without leading them. Later on you might switch over to a multiple-choice question that is informed by the results you gathered from the free-text exercise.
To 'mop up' any other feedback towards the end of a survey or section of a survey. This is where questions like 'Any other comments' work well.

Survey Structure

The layout and length of your survey can cause you problems if you get them wrong. For example, surveys that are too long can lead to survey fatigue, while surveys that ask demographic questions at the beginning can influence the way people answer your questions.

Keep it Short
The longer the survey, the more likely you are to get compromised results. People may just get fed up and abandon the survey part way through. Alternatively, they'll stop paying attention to the questions and simply click randomly just to get to the end. Either way, you'll end up with unreliable results.

An effective way to keep your survey short is to start with your 'why'. Why are you running this survey? What do you hope to learn from it? List what you want to learn before you start writing the survey. Then, at the end, you can compare the survey you wrote to the initial list. Remove any questions that don't meet your needs.

Avoid Stereotype Threat
Studies show that the order in which you ask questions can affect the results because of something called stereotype threat. This is when people conform to their social stereotype if they've been asked about it first.

For example, two groups of senior citizens take a mental agility test. One group is asked their age before the test. The second group is asked after the test. The group that was asked afterward perform better. The group that was asked their age before the test conforms to the stereotype of diminished mental capacity with age. Essentially the age question primes them to perform worse.

So how does this relate to your stream? It's simple: Either avoid asking demographic questions entirely or ask them at the end.

Effective Question Phrasing

Simple, Clear Language
You need people to understand your question instantly so they can move on to thinking about their answers. This means you should stick to clear, direct language.
Avoid long words, jargon, and words with multiple meanings.

Unsure whether a word is too complicated? You can check it against the 1,000 most used words in your language. There are lots of lists available online. One of them is https://1000mostcommonwords.com/.

Specific Language
Your question may mean different things to different people. Put yourself and their shoes and be as specific as you can when you write your questions. For example, consider the difference between "Do you exercise regularly?" and "How many times per week do you do more than 30 minutes of exercise?" The second version of the question is longer but it's also clearer. It is more likely to give you reliable results.

Smaller Questions
When using questions to pick at a theme, break your questions down into smaller pieces. For example, imagine you're a coffee shop asking about satisfaction. Don't just ask, "How satisfied are you?". You could also ask about:

Satisfaction with the flavor
Satisfaction with the service
Satisfaction with the environment
Satisfaction with the cleanliness

After all, you could ask the overall satisfaction question on your ExpressPod, with the follow-up questions presented via your stream Link.

Avoid Double-Barreled Questions
Each question should ask about only one thing. This sounds simple but it's easy to fall into the trap of writing double-barreled questions. For example, "How often do you eat fruits & vegetables?" can be hard for some people to answer because they might only eat fruits and not vegetables, or vice versa!

>You can check your stream for double-barreled questions by looking for words like "and", "or" in your questions.

Avoid Leading Questions
It's easy for your own opinions or biases to creep into your questions. These will tilt the results and make them less reliable. Consider the following three examples:

"Should we cut our summer party budget to increase our charitable donations to help victims of famine? (Yes, No)"
"Should we increase our charitable donations to help victims of famine? (Yes, No)"
"From which of the following areas should we redirect funds to our charities budget? (Followed by several options)"

If you were asked the three different versions of this question, how would you be influenced?

You can check your stream for leading questions by asking a colleague to review it before publishing. Look out for questions that seem as if there is a 'right' answer.

Summary

There is lots of information here, and even more available on the Web. If we were to boil it down to the five key lessons, they would be:

Start with why.
Use plain language.
Stick to one item per question.
Keep the survey short.
Look for bias before you publish.

Updated on: 11/08/2023