Surveys are a very commonly used tool on the internet. Both in marketing and social research they are used as a method to understand people’s behavior and preferences.Online surveys make it easy to launch, collect and analyze data and even combine data with behavior tracking, creating new grounds for understanding social data. Unfortunately most surveys are poorly designed, perhaps an effect of how easy it has become to create and launch them online. This post will focus on the pitfalls of designing and using online surveys. Here I will give some advice on the five main phases in the survey process:

1. Focus
2. Questions
3. Data quality and technical considerations
4. Design and Launch
5. Analyze and Compare

Focus

Most surveys fall prey to a complete lack of plan and focus. Perhaps nine out of ten surveys are created without any overall idea of purpose and strategy for using data. Or perhaps there are too many things which could be interesting, but really are not. Of course there are very often more than one person interested in how to collect data, what questions to ask and standards to meet and the process of planning can seem intolerable and unnecessary long. But generally one should consider using more time designing a plan for the whole process.This also includes considering exactly what answers to expect from the questions. An example of this is when more than one person are involved. Often this will lead to different persons asking different questions in the same questionnaire. The result is an abundance of data being difficult to analyze.What is the solution? Try to find the main idea of the questionnaire and skip all questions not having a clear connection to this idea and a strategy for analytically establishing this connection.
Try to think of it as a theoretical problem where one theory is put into question and then made more or less convincing using the knowledge obtained from the questionnaire.

Questions

Designing the right questions makes you vulnerable to many traps. If you read “Mail and Internet Surveys” by Don Dillman you will find answers to almost anything which has to do with asking questions. If you want to skip the 500 pages, you should a least look for the following common mistakes:

• One question per question!
  When you ask questions, don’t ask more than one question e.g. “Are you happy with your family and job situation?” unless it is possible to give more complex answers (which should generally be avoided) like “Yes, my family situation is fine, but I am busy at work” etc.
• Be interesting
  Always begin the questionnaire with an interesting and stimulating question like “Would you do X, if you could get a reward?” never just something like “What is your total household income?”. For the rest of the questionnaire, try to maintain a balance between interesting questions and boring id questions.
• Scales
  There are many different opinions about scales. How many categories, even or uneven number of categories etc.? Scientific research indicates that only few conclusions go uncontested. Firstly, it is very difficult to compare scales across different questionnaires. Try changing just some minor details and users will answers differently. There is only one solution to this problem. Be very careful to compare scales across different questionnaires and never try to compare questions from different worlds like online versus offline self-administered questionnaires. Secondly, the number of categories does not seem to matter. Just make sure there are not too many and always remember that respondents scan questions and categories.
• Randomize
  If possible always try to randomize questions. In this way you will avoid bias towards specific answers, notable the second category on the list (mostly preferred by respondents) and the first.
• Allow comments
  Always allow comments to the questionnaire at some point. Even if you are sure of what to ask, some respondents simply want to give feedback on your (to them) absurd or irrelevant questions. There is no reason not to and if you don’t learn anything by reading the answers at least you will probably have some fun reading them!
• Segment your respondents when asking
  It is always a good idea to try to break down your respondents into different types. Even if these types are very one-dimensional and simplified (like student, employer, customer etc.) it has great analytical value.
• Know your respondents
  Don’t give answers that make people angry. This will probably just give wrong answers in return. A recent example was a survey where respondents had to answers their yearly earnings. The first category was up to £100.000, the second £100.000 - £1.000.000, then £1.000.000-£10.000.000, then £10.000.000-£100.000.000 and the last was +£100.000.000! Respondents simply begin to give wrong answers just for the fun of it.

Data quality and technical considerations

Everyone who has conducted a survey knows about data quality problems. In online surveys you don’t always have much control regarding this issue, but at least consider the following. Firstly newer try to make questions mandatory, not even if you are using very specific multivariate methods demanding complete answers.

The main problem is that if users are forced to answer questions two things can happen. Either they simply give up and close your survey (which is quite easy with self-administered surveys) or other, perhaps even worse, things will happen. They simply fill out a wrong answer producing statistically noise in your data and making analysis a more difficult to use (what seems more reliable data actually leads to less reliable conclusions).

Secondly try to get a survey tool that can handle data. This seems like an easy task, but the internet is somewhat more complicated than most people think. A browser is not just a browser and sending data packets on the net live is something very complex when using a browser. Often data is lost in the process. Think about this problem. How many respondents do you have? If few, then you should certainly consider these technical problems in detail.

Do not make things more complex than necessary. Most people like making if-then conditions or skip rules inside a questionnaire and your survey tool should most certainly support this feature. But don’t abuse this feature. If you make too many rules it will be very difficult to get an overview and more importantly it will be very difficult to analyze data.

If you are using tracking (behavioral) data in combination with survey data, you should pay very close attention to the quality of content descriptions. Besides, the combination of asking questions and at the same time tracking real-time behavior “invisibly” is a great opportunity to identify real behavior.Don’t believe you can just ask what people have done in the past. Research indicates that answers to questions about behavior going more than seven days back in time are seriously flawed. Most people have watched movies never made, read books never written, studied paintings never painted and so on.

Design and Launch

Design clearly has an impact on answers. Unfortunately there is not much we can do about it except one thing: Be consistent and don’t be creative! Use simple pages with few questions on each, don’t use graphics beside the logo, use ordinary scales (highest value first, one row or column, no need to use numbers). And always remember that respondents rarely read all words.Think of every page as having a number of words at its disposition. No matter how much you write on the page, respondents will not read more than this number of words on each page (perhaps 50, perhaps 100 or perhaps even more but always dependent on the context). More words than this is just causing more confusion and possibly misinterpretation and bias towards specific unknown readings.There is one design feature that should never be forgotten: Place your logo on every page. Research has shown that using a logo (from a company, a university etc.) without doubt causes higher response frequencies. Don’t miss that opportunity.

Analyze and Compare

Thinking about your analytical methods before you begin designing questions is fundamental to your success. In multivariate analytical methods it is almost always necessary to design questions in a smart way if you want to produce interesting results. In this type of analysis, as well as in many others, you would want homogenous groups (according to size) which can only be constructed at the beginning of the survey process (the question design phase).When you have finally reached well designed good quality answers you can begin analyzing the questions, which can be done in too many ways to be covered in this post. But always remember that more complex methods are not always better and are often more prone to bias and poor design quality. Simple comparison inside the survey scope and to identical surveys is almost always rewarding – just remember to take great care when comparing your online with your offline data. Just one last reminder: Test your survey before it is too late!