Free QA! David Golden

Size: px

Start display at page:

Download "Free QA! David Golden"

Camilla Quinn
5 years ago
Views:

1 Free QA! David Golden OSCON July 22, 2010

2 Free as in beer Free as in speech

3 Quality + Assurance

4 Quality is an ideal

5 Quality Assurance is what you can measure

6 CPAN Testers is Free QA for CPAN The objective of the group is to test as many of the distributions on CPAN as possible, on as many platforms as possible. The ultimate goal is to improve the portability of the distributions on CPAN, and provide good feedback to the authors.

7 CPAN is 15 years young ~135,000 uploads from ~5000 authors ~23,000 unique libraries uploaded ~21,000 active libraries available ~1,000 uploads every month ~200 new libraries every month

8 CPAN Testers is 11 years young 7.8 million reports CPAN Testers Peak month: 400,000+ reports

9 Learning from Crisis

10 Four ideas for success

11 Culture Fun Mashups Scale

12 Culture Values Tools

13 $ perl Makefile.PL $ make $ make test $ make install

14 $ perl Makefile.PL $ make $ make test $ make install

15 CPAN (1995) CPAN.pm (1996) $ cpan File::Spec (runs 'make test')

16 We built tests into our tools

17 Thousands of people running CPAN tests... and throwing them away

18 CPAN Testers (1999) Reports submitted "by hand"

19 CPANPLUS (2002) Can submit reports "automatically" when you test or install

20 A step change in participation Reports/ Upload A. Tang, Becoming a CPAN Tester with CPANPLUS (2002)

21 We built tests and test reporting into our tools

22 But still just ~5 reports per upload Reports/ Upload

23 Crisis #1: Lack of coverage CPAN Testers no longer a toy but not yet useful for much

24 Fun Community Competition

26 Step 1: Make it easy to join

27 CPAN::YACSmoke (2005) POE::Component::CPAN::YACSmoke (2006) CPAN::Reporter (2006)

28 Step 2: Build a community and give feedback

29 (2006) (2007) Barbie starts a monthly blog (2007)

30 Competition

31 September, 2007 "The heat is on for Chris Williams, he only thought Dave Cantrell was chasing his heels, but Andreas Koenig has also stepped up his game too. It has been a staggering month for the sheer volume of reports..." -Barbie

32 Climb the leaderboard

34 Test more or weirder Perls and platforms

35 Perl version and OS coverage

36 The result of -Ofun Reports/ Upload

37 Zooming out Reports/ Upload

38 rise of the machines

39 CPAN::Reporter::Smoker (2008) POE::Component::SmokeBox (2008) CPANPLUS::YACSmoke (2009) minismokebox (2009)

40 Reports by #1 tester each month 1,000, ,000 Dedicated Smoking Smoke Clusters 10,000 1,000 Manual Reporting Automated Reporting

41 More testers & reports per tester Number of Testers 250 Smoke Clusters Automated Reporting Dedicated Smoking 50 Manual Reporting 0 Reports per Tester

42 Crisis #2: Spam ~15,000 reports a month (2006) ~250,000 reports a month (2008) By

43 2006: ~5 reports/upload Joe Tester Cc: To:

44 2008: ~120 reports/upload WTF?!? "Stop SPAMMING me!"

45 NOT fun for CPAN authors Disabled author CC's Notification through a central service Added author contact preferences

46 Mashups Innovation Insight

47 Open data allows innovation

48 Test matrix matrix.cpantesters.org

49 Dependency tree deps.cpantesters.org

50 "Volatile 100" "FAIL 100"

51 Regression analysis and failure correlations o o o OS, architecture, perl version, perl configuration test output, e.g. failed at line tester, test framework analysis.cpantesters.org but offline!

52 Crisis #3: overload Reports sent by Reports re-distributed via NNTP

53 Success? Failure? November 2009 perl.org mailing list monthly volume perl.cpan.testers 455,875 perl.cpan.uploads 2,080 perl.perl5.porters 1,163 perl.daily-build.reports 953 perl.beginners 555 perl.modules 525 perl.perl5.changes 206 perl.moose 117 perl.module-authors 95 perl.perl6.language 89

54 The way perl.org sees perl.cpan.testers everything else 99%

55 November 2009: throttling back

56 Scale Anticipation Leverage

57 Scale is not always a technical problem!

58 Visualize wild success

59 But don't build for it right away

60 worked! NNTP worked! For 10 years

61 1. Anticipate the breakpoint 2. Do something about it

62 FAIL We should have seen it coming sooner

63 We built 2.0 by designing an API for server components and implementing a version of it on AWS

64 Clouds give leverage

65 Technical scale Administrative scale

66 Volunteer time is more scarce than nominal cloud costs

67 CT 2.0 running in beta since April

68 Culture within values and tools Fun from community and competition Mashups spur innovation and insight Scale needs anticipation and leverage

69 Thank you for coming! Your QA task: Rate this talk, leave comments and tell me what could be better

Vanilla Perl. David Golden. Perl Seminar New York October 17, Copyright 2006 David A.

Vanilla Perl. David Golden. Perl Seminar New York October 17, Copyright 2006 David A. Vanilla Perl http://vanillaperl.com/ David Golden Perl Seminar New York October 17, 2006 Copyright 2006 David A. Do you use Win32? 1 Copyright 2006 David A. Do you use Perl on Win32? 2 Copyright 2006 David