Conducting a survey pilot is essential. Piloting the survey and the communications helps confirm the validity of your questions and the effectiveness of the messages in your communication. The survey pilot should identify questions, messages or instructions that don’t make sense, or will lead to biased responses. It can also be useful to identify areas that may have been missed when designing the survey.
Pilot the end-to-end process
The pilot should aim to cover, and get feedback about, the entire survey process, not just the survey questions. That includes the survey invitation, the instructions, questions, response options, messages, emails, reminder messages and the thank you messages.
Choosing the survey pilot group
At a minimum, you should first pilot the survey yourself. Then, consider piloting with the members of your project team. Beyond that, identify people from your target audience and engage them in the survey pilot. The number and nature of people to involve in the pilot will depend upon a range of factors, including the goals and objectives of the survey, the size of the taget audience, the diversity of the target audience, and the availability of pilot participants.
Two types of survey pilot are Participatory and Undeclared
In a participatory pilot the participants are aware that they are participating in a pilot. The pilot participants are clearly informed that they are participating in a survey pilot. This type of pilot is useful when feedback is needed from the target audience about the survey content, processes and messages.
In an Undeclared pilot, the participants are not aware that they are part of a pilot group. The survey is issued as if the survey were real. This type of pilot is useful to ensure there are no issues with survey completion or to review the results received.
Be clear about what is expected of the survey pilot group
What you communicate to the pilot group is likely to depend upon whether the pilot group are aware that they are participating in a pilot or not.
In a Participatory pilot, you should engage pilot participants in such a way that makes them want to be involved and to help. The pilot participants need to be aware of what is expected of them in the pilot process and how their pilot data will be used. More than likely, the survey will be refined after the pilot, so the responses from the pilot group will need to be deleted. The pilot participants need to be aware of this and that they will be asked to complete the survey again when it goes live.
These pilot participants need clear guidelines about 1) what they are to look for and 2) how they will provide feedback about the survey. E.g. can they provide their feedback in the live pilot survey? Do they have to take notes and send them separately? Is an online collaboration tool being used to collate pilot feedback? When are they expected to provide feedback?
In an Undeclared pilot you will want to conduct a survey with a small live group of target respondents without them being aware that they are participating in a pilot.
Refine the survey and the communications
Once the survey pilot has been completed, review the feedback and refine the survey, instructions, key messages and the communications. Depending upon the pilot feedback, it may be necessary to conduct a follow up pilot, particularly if major issues are identified.
Once satisfied with the survey, it is important to get sign-off by the project key stakeholder.