I saw a very nice 20-min video about user testing and would like to share with you : http://videolectures.net/kdd07_kohavi_pctce/
Title: Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO
I love the summary notes for user testing too - you will see this in the end of the video, but it's great that you can view the details of each point in the video.
- Listen to customers because our intuition at assessing new ideas is poor.
- Figure out the optimal solution based on data
- Compute the statistics carefully
- Experiment Often (triple your experiment rate and you triple your success rate & failure rate. Fail fast & often in order to succeed.)
- Create a trustworthy system to accelerate innovation
- Novice experimenters run 1% experiments
- To detect an effect, you need to expose a certain number of users to the treatment (based on power calculations)
- Fastest way to achieve that exposure is to run equal-probability variants (e.g. 50/50% for A/B)
- But don't start an experiment at 50/50% from the beginning: that's too much risk. Ramp-up over a short period
-- Yingying Zhang