Re: [ia-55] Share a nice 20-min video about user testing

From: Yingying
Sent on: Tuesday, April 24, 2012 10:12 PM
Thanks Aaron! Yes it's a nice video. And thank you for your notes!

I am kind of new to user testing, and there is one thing I was so impressed that he said - newbies are more likely to do fewer experiments ;)

On Tue, Apr 24, 2012 at 5:39 PM, Aaron Yoshitake <[address removed]> wrote:
Thanks for sending this, Yingying, I found it to be quite informative and instructive!

I especially liked Ron's characterization of data-driven decisions, which he contrasted with the usual process, i.e. decisions based on the Highest Paid Person's Opinion (HiPPO). 
While I'm job hunting I'll be sure to look for a company with a culture that respects data-driven decisions.

A couple things I'd add to your A/B testing notes:
  • "Monotonic ramp-up": During gradual ramp-up of an A/B test from 1% to 50%, keep users from the 1% "treatment" in the 2%, 5%, and other treatments - don't leave this up to chance.
  • Build a system that allows you to automatically abort the experiment if something goes wrong (i.e. page load time is much greater, JavaScript errors, etc.)
Thanks again! I highly recommend this video for anyone who's interested.

-Aaron

On Tue, Apr 24, 2012 at 12:20, Yingying <[address removed]> wrote:
Hi guys,

I saw a very nice 20-min video about user testing and would like to share with you :
http://videolectures.net/kdd07_kohavi_pctce/.
Title: Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO

I love the summary notes for user testing too - you will see this in the end of the video, but it's great that you can view the details of each point in the video.
  1. Listen to customers because our intuition at assessing new ideas is poor.
  2. Figure out the optimal solution based on data
  3. Compute the statistics carefully
  4. Experiment Often (triple your experiment rate and you triple your success rate & failure rate. Fail fast & often in order to succeed.)
  5. Create a trustworthy system to accelerate innovation


A/B testing
  • Novice experimenters run 1% experiments
  • To detect an effect, you need to expose a certain number of users to the treatment (based on power calculations)
  • Fastest way to achieve that exposure is to run equal-probability variants (e.g. 50/50% for A/B)
  • But don't start an experiment at 50/50% from the beginning: that's too much risk. Ramp-up over a short period

Enjoy:)

Yingying

--
Yingying Zhang




--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Yingying ([address removed]) from The Los Angeles User Experience Meetup.
To learn more about Yingying, visit his/her member profile
Set my mailing list to email me As they are sent | In one daily email | Don't send me mailing list messages

Meetup, PO Box 4668 #37895 New York, New York[masked] | [address removed]





--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Aaron Yoshitake ([address removed]) from The Los Angeles User Experience Meetup.
To learn more about Aaron Yoshitake, visit his/her member profile

Set my mailing list to email me As they are sent | In one daily email | Don't send me mailing list messages

Meetup, PO Box 4668 #37895 New York, New York[masked] | [address removed]



--
Yingying Zhang

Our Sponsors

People in this
Meetup are also in:

Sign up

Meetup members, Log in

By clicking "Sign up" or "Sign up using Facebook", you confirm that you accept our Terms of Service & Privacy Policy