Quite a while back Jacob Nielsen published on his site an article suggesting that 5 users are enough to uncover most of the problems in an interface when user testing. There has actually been quite a lot of research on this subject. Over at the measuringusability.com site by Jeff Sauro you can can read A Brief History of the the Magic Number 5 in usability testing to find a pretty well laid out chronology on the research.
One thing to note about Nielsen’s findings, if you read the whole article, he is not actually saying you should test with only 5 users just once. He recommends doing several series of tests with 5 users each time as you evolve your design. He also has recommendations on how many you should test if there is more than one user type/persona for the software or website. So it is better to take an iterative approach with your user testing rather than do a lot of users in one round. Formal user testing is expensive and you would be wise to structure your test plan to strategically test with the fewest number of users you need to make sure you get the best design for your money.
I think the main issue for determining how many users you test besides your budget is your organization’s capacity for incorporating changes based on the findings. It’s unfortunate, but true that some organizations will spend the time and effort to test with users and then turn around and do little or nothing to mitigate the findings. So before embarking on any user testing, it is very important to verify that the changes can be made regardless of what stage in the design or implementation process the testing is being performed.
If you are testing in the early stages with UX prototypes then hopefully you have included time in your own schedule to make changes to the UX specifications. Ideally you would do several rounds of testing as mentioned above as a best practice. If it is with a alpha or beta program in development, then you need to be sure that product owners and development are prepared and have left room in the schedule to work on changes based on findings from the usability sessions. In fact, if they have little time for changes then testing with only a few users is enough to uncover any major problems that you will have to advocate getting fixed.
Another great resource I found over on the measuringusability.com site is his explantation of the base equation for deriving a problem discovery sample size. There is an excellent table at the end of the article which cross indexes how many participants are needed for commonly accepted rates of probability you will uncover usability issues that affect a certain percentage of users. These are calculated using the formula N = log(x) ÷ log(p), where N is the number of participants needed and p is the probability of detecting those problems experienced by x percentage of users (if I understood it correctly). Using the formula yourself, you can set the probability threshold you are comfortable with for detecting problems that affect the percentage of users you think is significant for your application to get the specific number of users you will need.
Unless you are working on a high risk application, the most common usage is to be 80 to 85% sure of finding problems that affect 25 to 30% of users… Which works out to about 5-7 users as indicated by Nielsen. He also makes note that you would need representative users for each persona that you may have for a web site. Additionally, for early stages of UX design, it may be enough to test with only a few users in several rounds each, to work out issues with a design. Note this would still be a total of 9-12 different users as you would not use the same participants in subsequent rounds.
Generally, it seems it may be a waste of time and effort to test a single UI with more than 7 users at one time. The exception to this is of course when you need to test in multiple international regions or you are doing a summative test where you want to get as thorough as you can get in identifying as many usability issues as possible. Then you may want to crack out the formula above and determine what number you need… which ultimately will be affected by what you can afford and the capacity of your team to make changes as well.
Go to the comment form