You need a clear question to answer or a hypothesis to get before doing any evaluation. You also need to plan your study carefully, but however carefully you draw up your plans it is unlikely that you will anticipate everything that happens. Although good planning helps to ensure success it is not a substitute for experience, so do a small study, that is a pilot study, first before attempting your main study. Pilot studies provide an opportunity for you to learn from your mistakes without ruining your main study.
The possible benefits of a pilot study include:
The main study can be planned better because you will see what kinds of problems occur in the pilot and you can fix them,
It provides an opportunity to practice a technique so that data collection is consistent,
You will be more confident
For example, before giving out a questionnaire to 500 people it would be a good idea to give a draft of it to just 20 people to try out. From this pilot study, you would see which questions were ambiguous or poorly worded, enabling you to change the questionnaire before doing the main study. Distributing a poorly designed questionnaire to 500 people could be an expensive mistake if you later decide that the respondents were not able to answer one or two of the most important questions because they were poorly worded.
Users' rights
Don't forget that users are people too, and that they have other things to do apart from your tests, and that they have feelings. They may get anxious if they think you are watching what they do and perhaps criticizing them. Everyone feels threatened at the prospect of being thought stupid. Users generally do not have to help you, and, as well as remembering that yourself, it is good to make them aware of their own rights and to let them know that you are sensitive to these rights. In particular, you should guarantee them privacy and assure them that data about their interactions will not be released to anyone else. This means that you should develop some way of labeling transcripts of interviews, questionnaires, and other data. For example, you can use numbers instead of names. You should also make sure that your users know what the aim f the test is in advance, the kind of activity that you will ask them to do, and that they are free to withdraw from the test at any time.
Why pilot sessions are more valuable for international Design Research
We have already established that pilot sessions are of value by ensuring that the discussion guide achieves the goals before the research “proper” begins. Pilot sessions allow you to have confidence in the research approach and moderator in a way that observing sessions and offering feedback between them doesn’t. Feedback between sessions is hard to deliver and hard to take. Remote research makes this even harder and doesn’t help make the research more effective.
The researcher may already be stressed and the last thing they need is to feel their capability as a moderator is being criticized – even if it isn’t. Minor tweaks are OK but wholesale changes are very hard to make work effectively and in any case, you are losing sessions as you make those changes. For the client, it can feel like the moderator isn’t listening to the feedback. The moderator will have spent a couple of days really getting to grips with the discussion guide and last-minute changes don’t always stick. The client can become very frustrated about going over the same thing between every session.
A pilot session removes all this pain. Because feedback is being delivered in the context of developing the discussion guide it isn’t taken as personal. It can also be worked on, fine-tuned, and then added so is included in the final immersion. That makes sure the moderator knows exactly what they are doing and you are confident in them. Anyone who has watched remote research with an 8-hour time difference knows, you struggle to bring your a-game. Sometimes you can’t watch at all. Sometimes your team doesn’t show up, or the connection is poor. There are a whole host of reasons why you should plan to not be there and a pilot session is a primary solution.
Conclusion
If you are not already using pilot sessions to make your research better I urge you to reconsider. For very little additional outlay, if properly planned, they can deliver a huge amount of value. They save money, avoid wasting sessions, ensure objectives are met and that the moderator is very clear on what is required by the client. That is the case for single market research projects but even more so for international design research.
Photo by Dan Dimmock on Unsplash
Comments