Often one of the biggest challenges of conducting user research is not writing out proper tasks (although that is challenging as well), or setting up the space and equipment, or even running the actual sessions, it is finding the right participants. Finding the right someone to observe and give useful feedback on your application or web site can make a all the difference in getting the right information to make decisions on how to improve your product. In fact, observing the wrong people can lead you to the wrong conclusions about what may or may not be working on your site. The following are some helpful tips and recommendations for getting the right participants.
What and where are you testing
First determine what type of research you are doing and where you need to conduct the research. Will you need to do field studies, or usability testing? Will you need to go to the particpant’s location, invite them to a lab or conference room, or use an online research tool? Some methods of recruiting people are more helpful depending on the research you need to perform and/or where you need to do the research.
Create a screener
To get to the kind of people you want to observe you should put together what is called a screener script or survey. This script is used to screen out participants who do not fit the target group you are trying to reach. When creating your screener, you first need to know what kind of observations and feedback you need. For most consumer focused applications or web sites, in particular ecommerce sites, it may seem that anyone would be a good candidate for observing and providing feedback. However, if you are trying to increase the number of new people who use your site, then you may really want people who have never used your site before. Whereas, if you are adding a new feature to an already existing application/site, you may want to be sure to get people who have used that feature before to ensure the new feature does not break the existing experience for them.
You can put other things in the screener such as the size of the team they work with or what other applications or sites they use. They can help provide useful information about the context of use for your website/application. While you can also screen for personal demographics like age, experience, education, location, income, etc, I don’t recommend filtering out based on these aspects unless they are an integral part of the target group. Also, don’t waste time asking for very granular answers on the screener that do not have meaningful differences for participants or for your app/site. For example, is there really going to be a difference between people who are 50-55 years old and 56-60? And if there is, how is that going to impact your design decisions? It might be better to know simply if a person is actively working or retired.
Where to find people
Once you have a screener ready you can set about actually finding people. Some have recommended just going out to the coffee shop or public space and finding a few folks and quickly show them what you are working on. This may be fine if really anyone’s behavior is going to be representative of the people who you expect to user your app/site. If you take this approach it is likely you may not have time to go through a lengthy screener, so you may want to dispense with the full screener and only ask a few pertinent questions to identify likely users. However, coffee shop/totally random people can also easily end up providing a lot of misleading observations from people who may never use your site or application even though they may fit your target. The most appropriate use of completely random users is for testing what should be basic universal interactions or when you are trying to reach those potential customers who might benefit from using your app/site, but simply have not yet tried finding a solution to their need.
Another place you can find participants, is within your existing customers. This is great for getting people who already use your site/application. This can work well for both in-person and online research activities. I would prefer this as well for ethnographic, and field studies research methods. Again, this is most useful when needing feedback on new and/or improved features. However, be careful of lists provided by internal customer service or sales teams as these tend to fall into two groups, those who are very good customers already and those who have a lot of complaints. The main problem with both, is they are perhaps over familiar with your business.
The good customers may accept some of the broken things and may not even think about them as problems since they are so used to the existing UI. While it is good to get their feedback to ensure you have not broken the existing experience in a more critical way for them, they may also give false positive responses to things that are different, but not actually easier for them. The complainers may be looking for things that are completely unique to them and would be of no benefit to the rest of your target group. They do this by providing false negative responses to things that are improved, but they are intent on complaining about other issues as well.
You hopefully have already seen the most common complaints through your customer feedback system (assuming you have one) and incorporated those in the design. Having these frustrated users try an newly designed version is great, but being given suggestions to add an esoteric feature that won’t even increase their usage of your application/site is just adding to the noise. To be clear, there may be some really good features suggested by complainers, but there may be little or no actual improvement in the experience for the greater number of people you want to target despite how awesome the feature sounds.
Another way to get existing and new customers is from your site itself. You can put a banner or link on your own site asking users to provide feedback. You can then take what is your screener and turn it into an online survey. At the end you can ask if they are willing to participate in further research and ask for appropriate contact info. The benefit of taking this approach is you can begin to build a pool of potential participants for on going research.
Lastly, depending on the nature of the business, using existing customers may not work well for testing a web site, if they do most of their business off-line. For multi-channel businesses, while there will be some overlap in customers using more than one channel, be careful of recruiting customers for whom the off-line channel is preferred. It may be your desire to convert those customers to more profitable online channels, but there may be a powerful preference for off-line interactions.
If you are targeting a specific profession or type of participant and you want people who have not used your product before, you will most likely need to get a recruiter involved. As noted previously, there may be some situations where you can seek contact information for existing users/customers, but this can be challenging when an organization’s sales or support teams want to control access to that list and only provide you customers who they want to try and sell more to and/or impress. In some situations they may not even want to share that customer info as they worry about what you may say to them. Going through a recruiter will allow you to screen for a mix of existing customers and potential new customers.
The main difficulty with using a recruiter is the expense. You may need a substantial incentive as well as pay a fee to the recruiter for each participant. When recruiting for certain professions this could be up to $300-500 USD per person for this incentive and fee combined. Generally, I would use a recruiter only for in-person usability testing, card sorting, or other research method where you want to meet the participant. While recruiters may offer lower fees for having participants attend additional later sessions (i.e place them on retainer), this is not recommended as they may become familiar with your designs and your moderator.
Industry Trade Publications
Another way of reaching specific professions, is through a mailing list that you can obtain for a fee from an industry/trade publication or organization. You can convert your screener to an online survey and then send an email out to the list soliciting folks to participate in the survey. At the conclusion of the survey you can then ask for appropriate contact info for your research needs. This works well for both in-person and online research activities. A list from one of these sources will generally run $5,000 or more depending on the industry and size of list, which may be beyond most budgets. However you may want to check with your internal marketing team as they may already have a list (or a budget). You can then add a small call to action to a series of their weekly or monthly e-mail campaigns that again links to your online screener survey.
The Right Participants Provide the Right Insights
In closing, you should be aware of what types of participants you need when doing research. Observing and getting feedback from the wrong type of participants can lead you to make changes that have a negative impact and/or add unneeded features to your product or service. Creating and using a screener script/survey will help filter out people who would likely not use your product or service. Getting great observations and unbiased feedback can be a challenge and sometimes expensive, but it is worth it as you will gain truly useful insights with observations and feedback from potential and/or existing customers who represent your actual target audience.Go to the comment form