Survey Design 101 (How to plan and design a perfect survey)

Surveys, also known as questionnaires, have become a key tool for gathering valuable information regarding any topic. To succeed in doing so, there are a few best practices you can use to drastically improve the quality of data.

How to achieve a good survey design?

First, let's take a look at the core of a good survey blueprint.
It is essential to project a defined mental image of the audience you want to survey. Once that is set, it's time to begin developing the questions. A rule of thumb to follow when aiming to write them is to start with broader, less specific parts and then progressively shift into more particular, precise questions. At some point, you will be gazing at a list of "finished" questions. It's not yet time to start the survey, but rather to initiate the following step: pilot testing and rewriting. Essentially, this part of the process is based on trial and error. Select a few testers and keep polishing the questions until you are thoroughly satisfied with them.

In conclusion, achieving the best possible survey design is ultimately leading you towards collecting the highest quality answers that your study needs to become trustworthy. Although at plain sight a questionnaire can look like merely a compendium of questions, elements like its presentation or cohesion among parts are relevant enough to keep in mind, as they're going to boost your results.

In the following paragraphs we're going to dive deep into everything you need to know on how to plan and conduct a successful survey. Keep reading to learn everything about how to prevent key mistakes, how to define your audience and create suitable questions while offering them reasons to participate in the study, and how to pilot test until the survey is ready.

Good survey design equals better results

Moving on, let's shed some light upon why we believe a good survey design is so crucial. In essence, three fundamental issues need to be eliminated during the planning phase.

Response Bias - this term refers to all of the tendencies behind each false or inaccurately answered question. Many factors such as misleading questions or private topics that haven't been appropriately introduced can lead to a lower rate of honesty by respondents. Since your currency (and what makes your work valuable) is the ability to rely on the data you collect, aiming to reduce response bias is ultimately going to enhance the quality of your survey.
The first technique you can employ to minimize the bias is making sure your topic is properly identified and you (and your audience) fully understand it. Utilizing the pilot group as a tool to carry open discussions is a way to make sure all questions and predefined answers are clear to your audience, as we'll see soon.
Next, choose between open-ended and closed-ended questions. Each option has its own set of pros and cons, meaning it's up to you to decide which one suits your survey best. While open-ended questions are better for more broad topics where answers may vary significantly and are hard to concisely display, closed-ended questions are much easier to analyze and to answer, leading to a higher response rate. The downside is that you need to be thorough and precise with your questions, as they will represent the voice of your audience.
In terms of bias, the open-ended ones will secure a lower rate of misleading information, as it allows for a more accurate depiction of the respondent's thoughts.
Terminology inconsistency - as mentioned above, cohesion of your questions and answers is fundamental. For example, on the terms used. Following this rule, if you employ the word "check", don't alter it halfway through the survey for "mark" or any other synonym. Remain loyal to specific terminology. The same thing applies to design, as it is the whole point of the article. The graphs and charts must respect a graphic order, as anything besides that would create a sense of chaos and flawed structure that eventually diverts the respondent's attention.
Low engagement and finish rate - when it comes to design, a clean and defined structure is going to guarantee higher finishing rates. One of the principal guidelines to follow is to avoid overloading the survey with complicated questions. Keep it simple and concise. When you mix this with appealing writing and visual design, expect significant improvements when it comes to the finishing rates. Also, if you want to get a better response rate, you need to make sure that your audience is motivated enough to finish your survey.

Survey audience

As stated before, the ultimate goal of a survey is to warrant that the data you collect matches the true thoughts, feelings, and interests of the people that answer it. With that in mind, what should you do to ask the right questions?
Write, try and repeat. The pilot testing technique is indispensable. However, one thing you ought to do is spend time researching your audience. Get to know them while you keep asking the questions to yourself. Once you grasp their core identity, it will be much easier to identify meaningful questions from those who contribute nothing. Put on their shoes and try to be as critical as possible.

On the other hand, how can you recruit respondents? There are many ways to gather people. You could share the word through social media, email newsletter, website sticky bar, you can even buy ads to drive the audience to your survey. All these things need to be taken into consideration while planning a survey. Also, several trustworthy services were created to provide a solution for both people interested in carrying out surveys and those who are consistently seeking respondents. Some examples could be "Call for participants" or "Survey Circle". They can help you to find respondents faster. However, it’s not necessary to invest money to find respondents.

Besides, a big part of the recruitment phase comes down to common sense or intuition. You wouldn't look for rich people in the neighborhoods with the lowest income, as you wouldn't try to sign up people for an upcoming video game survey in a conservative political rally. Study your audience and implement the habit of having open conversations with your designated pilot testing team before the definitive survey to maximize the reliability of your results.

What do the respondents exchange their time for?

In today's society, it is no secret that the attention span of the average individual is worryingly short. This may be a consequence of the evergoing media bombing we live under, exploiting the fear of missing out, as well as the need to be updated on every single subject.
Given this tendency, it has become an arduous task to recruit people and keep their attention for as long as the survey needs. This is why you should give them a reason to participate.

While you may think everyone is merely going to feel instantly appealed towards your project, the reality might be harsher than that. Do not dismiss any potential opportunity to add value to your survey. What can you offer?

First of all, money. Despite most surveys being free, if the sample you're seeking is especially low or hard to convince (or perhaps they could use some extra money), consider an economical reward. Note that this method won't probably work when surveying wealthy respondents.

If you cannot afford to pay or you simply do not wish to carry out things this way, think about what else you can bring to the table. Knowledge or self-knowledge are both suitable options, as you're providing valuable resources in exchange. This applies to surveys that teach about a subject or lead the person on a little introspective journey. Besides, presenting the survey as a quiz (though you should not conceal the nature of the survey) might make it easier for the respondents to feel engaged and willing to finish it, even if it's just because of the results at the end.

Another option is to appeal to the inherent benevolence of the human heart, also known as "Asking for help". Share the word through social media or in person, as mentioned above, and make sure everyone who might be interested hears about it. There are still people out there willing to follow the path of selflessness and cooperation just for the sake of it.

What to ask and when to do it?

As we've seen so far, how each question is displayed is just as relevant as the question itself. Avoiding grammatical mistakes and cohesion inconsistencies are moving you close to a successful survey before any piece of data has been collected.

Let's start with the order your questions should follow.

Again, these are just guidelines. Each survey is unique and requires a different approach. Think of this as the skeleton on which you're building the body around. The sequence of the questions usually wants to start with the broader side of the subject, as well as the formalities. The beginning is the place to describe and explain your survey and its purpose, as well as ask for the name, age, and any other mandatory information. Once completed, the real questions begin.
Widen your scope and slowly narrow down as the questionnaire advances, especially if you're including any sensitive question: leave those for last.

Before providing more in-depth information about the types of questions, it's helpful to list some of the most common mistakes you should keep in mind and stay away from.

Leading questions: The respondent should never feel like they're being pushed or pulled towards a specific answer. The writing must be neutral to avoid any kind of pressure on them, as it can reveal the results you are expecting to get from the survey.
Double-barreled questions: Always check that none of your questions are two or merged. Put them separately, given it can become confusing and end up in more complex data to analyze.
Mixing question types: A structure that shouts disarray is never going to achieve the results you expect. Be tidy and meticulous, as people need time to adapt to the change of question types. For example, mixing open-ended questions with closed-ended ones. Keep in mind that the time they're willing to dedicate to you is limited, meaning you show respect for their commitment and shouldn't push it further than necessary.
Prestige-bias: Your goal is to collect and analyze data that is as close to the truth as possible. It is not to mention your preferences or referrers. The person answering the questions shouldn't evoke anything that's not essential to the survey.

Next, the most usual types of questions.

Closed ended: this variety excels in making the collection of the results as easy as possible. They can be Multiple Choice (picking from a variety of options), Multi-select (one, two, or more answers "checked"), Ranking Order, also known as Likert Scale (arranging the options from first to last or better to worse), and Rating Order: (sequence the options by score).
Open-ended: whilst being more genuine and allowing for more expression than its counterparts, open-ended questions are harder to analyze. This is caused by the unlikeliness of any answers being identical. An example could be: "What improvements would you suggest to the board of directors?", as it leaves room for each individual to speak their mind. Now, as an example, imagine there's a closed-ended question about the potential improvements that we just mentioned. You would want to put one question right after the other. The thing is, which one goes first? To diminish the level of influence and sense of leading, the open-ended question should come first. If you were to do it the other way around, the options might harm the following question, as the respondent might think of the options as clues for a "valid answer".

Again, before explaining what to do after you have finished your questions, remember: Keep your questions brief, clear, and simple. Create a sense of structure and cohesion and avoid jumping between different types of questions.

Take a break before launching your survey

Just like in pretty much anything in life, taking a break and separating yourself from your work for a while can be as helpful as hours of productivity. Specifically for surveys, there are some key moments where a break is going to kill two birds with a stone. Not literally, though. Birds are cool.

Those spaces of time can be double as helpful are the points just after you have completed a part of it. For example, whenever your initial draft of questions is laying on the table or your computer screen, save the document and stay away from it at least until the next day. On the next day, you will come back refreshed with a new pair of eyes. Again, not literally.

Just like when it comes to working as a team, where each perspective can be unique and helpful in its way, presenting the list of questions to different "you"'s is going to open new options and paths that you had not thought about until then. Moreover, the distance granted by the time away from the project is instantly going to provide you with the ability to look at it with a sharpened sense of constructive criticism, as if it was someone else's work.

Needless to say, burning yourself out is under no circumstances the way to approach any work or life project. Rest, take a break, go out, eat healthily, drink water, and get some sleep.

This mindset is what will bring you the capacity to reshape and polish your questions until you come up with the definitive list. But wait, it is not definitive at all, is it? No. As we mentioned, new eyes on the subject are crucial to look for improvement margin. It's time to assemble the team of your choice. It's time for the pilot test.

The Beta phase of every good survey

A videogame reference seems adequate here since one of the last steps before a game goes "Golden" (slang for finished) is the Beta phase. In this development stage, the goal is to polish the product from all of its imperfections and undesired interactions. When translated to survey language, the pilot testing becomes the period in which you aim to bring your questions as close as possible from the good survey design foundations that we talked about earlier.
To complete such a task successfully, there are a few things to keep in mind to ensure you're getting as much value as humanly possible from the beta testers or pilot testers.

A proper pilot testing team should resemble a random sample of people taken from those who would answer your survey. This means they should fit the demographic background profile the questions aimed to. The dynamic should start with you delivering the questions and answer options to them to get all observations they can provide to improve the survey. Once that is done, it's time to review and repeat. Modify some questions, delete that one that you didn't feel was useful, correct some mistakes and give it another try. Take their feedback seriously, but don't make changes too lightly.

As mentioned above, proposing an open discussion with the pilot testers can additionally become an extremely valuable tool, especially to find out if your questions are gathering the data you intended and to ensure they don't come across as confusing. Remember that the ultimate goal of a good survey is to gather and analyze data that is as close as possible to the reality of the respondents' thoughts, feelings, and interests.

After as many repetitions you feel the questions need before being fully polished, it's time for the last step: the real world. Making good use of the survey design techniques listed above and applying them consciously is going to provide you with some vital support on attaining your goals.

Remember, survey design requires continuous refinement. Do not let setbacks let you down, it's never too late to gather some testers and begin the process once again. This path isn't a bed of roses, but the results can be as gratifying as it gets.

So, on a scale from "It changed my life" to "I wish I hadn't read it", did you find this article useful? Good luck!