Lecture Notes — Class 1 (pdf) - Claus Aranha @ Tsukuba University

Introduction What is Science? Bad Practices, Good Practices ConclusionExperiment Planning and DesignLecture 1: Introduction 1Claus Aranhacaranha@cs.tsukuba.ac.jpDepartment of Computer ScienceJune 7, 20131 version 1.1

Introduction What is Science? Bad Practices, Good Practices ConclusionGoals of the course

Introduction What is Science? Bad Practices, Good Practices ConclusionOutline of the courseFirst Class (Today)Fundamental ideas and concepts;Why should we care about Scientific Methodology?TomorrowBasic Statistical Techniques;Simple R tutorial;WednesdayMore Interesting Statistical Techniques;Working with Examples;

Introduction What is Science? Bad Practices, Good Practices ConclusionOutline of the courseFirst Class (Today)Fundamental ideas and concepts;Why should we care about Scientific Methodology?TomorrowBasic Statistical Techniques;Simple R tutorial;WednesdayMore Interesting Statistical Techniques;Working with Examples;• Topics may move around, depending on time;

Introduction What is Science? Bad Practices, Good Practices ConclusionOutline of the courseFirst Class (Today)Fundamental ideas and concepts;Why should we care about Scientific Methodology?TomorrowBasic Statistical Techniques;Simple R tutorial;WednesdayMore Interesting Statistical Techniques;Working with Examples;• Topics may move around, depending on time;• Feel free to make questions during the class!

Introduction What is Science? Bad Practices, Good Practices ConclusionGradingReportThe Goal of the course is to do an scientific and statisticalanalysis of your own research work.• Define the methodology used to create your experimentsand research goals;• Identify problematic parts in your methodology, andsuggest solutions;• Perform some basic analysis on your data;(If you don’t have a research topic yet, we can find a sample inthe last class)

Introduction What is Science? Bad Practices, Good Practices ConclusionGradingReportThe Goal of the course is to do an scientific and statisticalanalysis of your own research work.• Define the methodology used to create your experimentsand research goals;• Identify problematic parts in your methodology, andsuggest solutions;• Perform some basic analysis on your data;(If you don’t have a research topic yet, we can find a sample inthe last class)• Deadline – 2 Weeks after the last class (July 7th)• More details on the last class

Introduction What is Science? Bad Practices, Good Practices ConclusionWhat is science?• What is Science?

Introduction What is Science? Bad Practices, Good Practices ConclusionWhat is science?• What is Science?• What do Scientists do?

Introduction What is Science? Bad Practices, Good Practices ConclusionWhat is science?• What is Science?• What do Scientists do?• Are these the same question?

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.There are problems with this approach:

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.There are problems with this approach:• Experiments only show the system works;(of course it works)

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.There are problems with this approach:• Experiments only show the system works;(of course it works)• Reinventing the wheel;(why do I care about other methods?)

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.There are problems with this approach:• Experiments only show the system works;(of course it works)• Reinventing the wheel;(why do I care about other methods?)• No knowledge is actually acquired;(unreplicable papers, excess of data)

Introduction What is Science? Bad Practices, Good Practices Conclusion“Aperture Science”A common misunderstandingScience is about creating cool gadgets andrunning many experiments.There are problems with this approach:• Experiments only show the system works;(of course it works)• Reinventing the wheel;(why do I care about other methods?)• No knowledge is actually acquired;(unreplicable papers, excess of data)Is this really Science?

Introduction What is Science? Bad Practices, Good Practices ConclusionThe Harry Potter Game (1)What is Science?• There is a rule for triplets of numbers;• Some triplets obey this rule, some don’t;• You can ask me if a triplet obeys the rule, as many timesas you want;• The triplet 3, 6, 9 obeys the rule.What is the rule?

Introduction What is Science? Bad Practices, Good Practices ConclusionThe Harry Potter Game (1)What is Science?• There is a rule for triplets of numbers;• Some triplets obey this rule, some don’t;• You can ask me if a triplet obeys the rule, as many timesas you want;• The triplet 3, 6, 9 obeys the rule.What is the rule?Is that your final answer?

Introduction What is Science? Bad Practices, Good Practices ConclusionThe Harry Potter Game (2)The Rule

Introduction What is Science? Bad Practices, Good Practices ConclusionThe Harry Potter Game (2)The RuleThree Real Numbers in increasing order.

Introduction What is Science? Bad Practices, Good Practices ConclusionThe Harry Potter Game (2)The RuleThree Real Numbers in increasing order.• What strategy did you use to figure out your answer?• Did you try to confirm, or refute, your guesses?• Did you take other people’s guesses into account?

Introduction What is Science? Bad Practices, Good Practices ConclusionHans, the Smart HorseVon Osten, the horse’s trainer, displayedHans amazing ability to do mathematicalcalculations by tapping his hoof.

Introduction What is Science? Bad Practices, Good Practices ConclusionHans, the Smart HorseVon Osten, the horse’s trainer, displayedHans amazing ability to do mathematicalcalculations by tapping his hoof.Oskar Pfungst’s Experiments• Use people other than Van Osten to ask questions;

Introduction What is Science? Bad Practices, Good Practices ConclusionHans, the Smart HorseVon Osten, the horse’s trainer, displayedHans amazing ability to do mathematicalcalculations by tapping his hoof.Oskar Pfungst’s Experiments• Use people other than Van Osten to ask questions;• Put people in different places;

Introduction What is Science? Bad Practices, Good Practices ConclusionHans, the Smart HorseVon Osten, the horse’s trainer, displayedHans amazing ability to do mathematicalcalculations by tapping his hoof.Oskar Pfungst’s Experiments• Use people other than Van Osten to ask questions;• Put people in different places;• Make people ask questions they didn’t know the answer;

Introduction What is Science? Bad Practices, Good Practices ConclusionKoko the GorillaCan Koko use “Language”?

Introduction What is Science? Bad Practices, Good Practices ConclusionScience!Science is more than simply hapzardly performing experiments.Careful consideration of what are you doing is required.“Science is about being wrong”

Introduction What is Science? Bad Practices, Good Practices ConclusionWhy should I care about this?“I don’t care about testing my research - Ihave a very practical problem that I want tosolve.”• Methodology is necessary if you want to publish (in goodjournals);• You want to know if you are actually in the right path;• You want people to be able to repeat your experiments in thefuture;• Maybe you will find something you didn’t expect in the data;

Introduction What is Science? Bad Practices, Good Practices ConclusionA Science of Algorithms and Systems (1)• In “wet sciences”, we are usually interested in knowinghow something works.• In Information Sciences, a large part of our effort is spentcreating new systems.Can we conciliate “creating cool stuff” with good science?

Introduction What is Science? Bad Practices, Good Practices ConclusionA Science of Algorithms and Systems (2)Scientific questions for technological research:• How does the system work?

Introduction What is Science? Bad Practices, Good Practices ConclusionA Science of Algorithms and Systems (2)Scientific questions for technological research:• How does the system work?• DO the system REALLY work?

Introduction What is Science? Bad Practices, Good Practices ConclusionA Science of Algorithms and Systems (2)Scientific questions for technological research:• How does the system work?• DO the system REALLY work?• Is this system better than what we have?

Introduction What is Science? Bad Practices, Good Practices ConclusionA Science of Algorithms and Systems (2)Scientific questions for technological research:• How does the system work?• DO the system REALLY work?• Is this system better than what we have?(usually the most desired answer)

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Analysis of AlgorithmsNew Algorithms can be analysed from a theoretical orexperimental point of view.• Theoretical Analysis: Algorithm is broken down into itscomponent parts, and a mathematical model is built todetermine its behavior.Theorems, Proofs, Assymptotic Analysis;• Experimental Analysis: Algorithm is run on arepresentative sample of situations, in order to observe itsaverage behavior.Test runs, Benchmarks, Statistical Analysis;

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Analysis of AlgorithmsNew Algorithms can be analysed from a theoretical orexperimental point of view.• Theoretical Analysis: Algorithm is broken down into itscomponent parts, and a mathematical model is built todetermine its behavior.Theorems, Proofs, Assymptotic Analysis;• Experimental Analysis: Algorithm is run on arepresentative sample of situations, in order to observe itsaverage behavior.Test runs, Benchmarks, Statistical Analysis;Ideally: Theoretical + Experimental Analysis

Introduction What is Science? Bad Practices, Good Practices ConclusionBad PracticesCareless Experimentation“To Consult the statician after an experiment isfinished is often merely to ask him to conduct a postmortem examination. He can perhaps say what theexperiment died of.” – Sir Ronald Fischer• People are well meaning (I hope!), but sloppy research stillgets done.

Introduction What is Science? Bad Practices, Good Practices ConclusionBad PracticesCareless Experimentation“To Consult the statician after an experiment isfinished is often merely to ask him to conduct a postmortem examination. He can perhaps say what theexperiment died of.” – Sir Ronald Fischer• People are well meaning (I hope!), but sloppy research stillgets done.• Lack of training, Lack of awareness, Publication bias;

Introduction What is Science? Bad Practices, Good Practices ConclusionBad PracticesCareless Experimentation“To Consult the statician after an experiment isfinished is often merely to ask him to conduct a postmortem examination. He can perhaps say what theexperiment died of.” – Sir Ronald Fischer• People are well meaning (I hope!), but sloppy research stillgets done.• Lack of training, Lack of awareness, Publication bias;• Correct Goal: Finding out the truth;• Common Goal: Finding out that the algorithm ispublishable;

Introduction What is Science? Bad Practices, Good Practices ConclusionCareless Experimentation“The presented results represent the average from 30 runs ofeach algorithm”

Introduction What is Science? Bad Practices, Good Practices ConclusionCareless Experimentation“The presented results represent the average from 30 runs ofeach algorithm”What “average of 30 runs” does not sayProposed AlgorithmComparison Algorithm• Careful implementation anderror checking;• Exhaustive search ofappropriate parameters;• Many Experimental runs;• Careless implementation:Good enough if it runs;• Parameters found in theliterature;• Single experimental run;

Introduction What is Science? Bad Practices, Good Practices ConclusionCareless ExperimentationOther Common Problems• No clear definition of the Goal of the experiment (confusionamong hypothesis, theory, explanation, conjecture)• Cherry picking of data/results (This data that does not workfalls outside the scope of this paper).• Personal Biases (getting experimentation backwards:trying to show that one method is better, instead ofinvestigating if the method is better).

Introduction What is Science? Bad Practices, Good Practices ConclusionCareless Experimentation

Introduction What is Science? Bad Practices, Good Practices ConclusionOther common mistakesPremature conclusions• Stop as soon as a successful result is achieved;• No random repetitions, no stress tests;• Even some negative results can be informative!

Introduction What is Science? Bad Practices, Good Practices ConclusionImportant Principles for experimental research onalgorithms• Relevant Experiments;• Relation to the literature;• Significant test problems;• Experimental Design;• Efficient Implementation;• Reproducibility;• Comparability;• Complete Description;• Claims supported byresults;• Data properly presented.

Introduction What is Science? Bad Practices, Good Practices ConclusionPre Experimental Design (1)Before we begin• Do you have all knowledge needed?• Very common in information sciences• Specific factors/problems in the specific application• Teamwork

Introduction What is Science? Bad Practices, Good Practices ConclusionPre Experimental Design (1)Before we begin• Do you have all knowledge needed?• Very common in information sciences• Specific factors/problems in the specific application• Teamwork• Relevance of the experiment• Think about the target audience;• Check the literature throughly/avoid reinventing the wheel;• Is the proposed question relevant?• Break down the problem if necessary;

Introduction What is Science? Bad Practices, Good Practices ConclusionPre Experimental Design (2)Decide what is the question you want to answer:Objetives of an experiment• Determine the most influential variables in a system;• Determine desired parameter values to reach an output;• Describe the behavior of a system;• Etc.

Introduction What is Science? Bad Practices, Good Practices ConclusionPre Experimental Design (2)Decide what is the question you want to answer:Objetives of an experiment• Determine the most influential variables in a system;• Determine desired parameter values to reach an output;• Describe the behavior of a system;• Etc.(some) Types of Experiments• Time-series analysis;• Observation of the system;• Planned interference;

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (1)Parameter/Factor Selection• Educated Guesses• One Parameter at a time• Factorial Planning• Arbitrary combination of parameter/values for study;• Observe the behavior; Change some parameters; Repeat;• Risks premature convergence (“good enough”);Requires solid knowledge about the underlying problem.

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (1)Parameter/Factor Selection• Educated Guesses• One Parameter at a time• Factorial Planning• Selects and changes one parameter at a time;• Often used, safe;• Unable to determine parameter integration;

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (1)Parameter/Factor Selection• Educated Guesses• One Parameter at a time• Factorial Planning• Testing the influence of different factors on the system;• Values of each factor are varied in an organized fashion;• Determines individual effects and iteractions;• Experiment size increases with number of factors;

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (2)Data Selection• Benchmarks• Natural Data• Random Data

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (3)Design Principles• Repetition• Randomization• BlockingRepetition:• Repetition of the experiment allows for estimation of the error• Used for calculating statistical significance• Allows for a more precise measure• Decide beforehand the number of repetitions!

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (3)Design Principles• Repetition• Randomization• BlockingRandomization:• Random order of the experiments, Random allocation ofresources;• Reduce the bias from data, reduces influence from unrelatedfactors;• Where randomization is not possible: partial randomization,blocking, etc;

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (3)Design Principles• Repetition• Randomization• BlockingBlocking:• Removes the influence of unrelated factors• Breakdown experiments into blocks based on these factors• Reduces the number of available observations

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (4)Comparisons• Selecting Comparison methods• Recent Methods;• “Traditional” methods;• Methods Outside your discipline• Tweaking of Code and Parameter

Introduction What is Science? Bad Practices, Good Practices ConclusionExperimental Design (4)Comparisons• Selecting Comparison methods• Recent Methods;• “Traditional” methods;• Methods Outside your discipline• Tweaking of Code and Parameter

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (1)• A good experimental design should give you enoughinformation to select the correct statistical model.(Or close enough)• Export all possible data from your experiment, use knownstatistical tools to deal with the data(Programmers have a tendency to reinvent the wheel)• R, Matlab, Octave, etc...

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;2 Determine P(data|null model)

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;2 Determine P(data|null model)3 Decide Whether or not to reject the Null Hypothesis

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;2 Determine P(data|null model)3 Decide Whether or not to reject the Null HypothesisMany people stop here ...

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;2 Determine P(data|null model)3 Decide Whether or not to reject the Null Hypothesis4 Validate the premises of the Statistical Model

Introduction What is Science? Bad Practices, Good Practices ConclusionStatistical Analysis (2)General Procedure for Hypothesis Testing1 Define the Null Model and the desired significance level;2 Determine P(data|null model)3 Decide Whether or not to reject the Null Hypothesis4 Validate the premises of the Statistical Model5 Estimate the magnititude of the differences

Introduction What is Science? Bad Practices, Good Practices ConclusionAnalyzing the resultsStick to the Facts!Beware of overgeneralizing the results and making claims notsupported by the data.Stick to ALL the factsAvoid cherrypicking the results to make the system look betterthan it is. Complete descriptions are actually more useful.Dealing with anomaliesAny anomalies must be reported, but beware of “anomalyhunting”.

Introduction What is Science? Bad Practices, Good Practices ConclusionWriting that paper (1)Be ThroughReport everything that can help the reader understand theextents and limitations of your experiments:• Stop criterions;• Computational costs (if time, how it was measured);• Parameters, parameter selection methods;• Experimental setup (and its reasoning);• Etc;

Introduction What is Science? Bad Practices, Good Practices ConclusionWriting that paper (2)Strive for replicability!• Be very explicit describing your data• Cite literature benchmarks;• Algorithms/parameters for artificial data sets;• If hard to obtain, consider distributing your data online;• If at all possible, distribute the source code for yourexperiment!make sure you add all information needed for compilation...• Distribute online experimental data that did not fit thepaper;• Consider open access models of publication;remember you are usually allowed to distribute pre-prints

Introduction What is Science? Bad Practices, Good Practices ConclusionSummaryScience is more than cool gadgets, collecting data, and cakewhich is a lie anyway• Designing experiments requires adherence tomethodology;• Consider carefully the aims of the research and goals ofthe experiment;• Use of statistical methods and tools;• Beware of explicit and implicit biases;• Much of this is already done in other areas (Bio, Med).Watch and learn;“However, experiments require a lot of work, so the readermay be warned: performing a good experiment is asdemanding as providing a new theorem.”– Hans-Paul Schwefel

Introduction What is Science? Bad Practices, Good Practices ConclusionOne final messageIt is important to be literate in the scientific method, not only foryour own research’s sake. We are also agents of change in thepopulation and, as such, we need to be aware of good and badscience, and able to point the difference to the society.

Introduction What is Science? Bad Practices, Good Practices ConclusionOne final messageIt is important to be literate in the scientific method, not only foryour own research’s sake. We are also agents of change in thepopulation and, as such, we need to be aware of good and badscience, and able to point the difference to the society.Mini Homework:Look for “scientific” claims around you next week (news,advertisements, papers). Describe them in the next class, fromthe point of view of correct and incorrect scientific methods.

Introduction What is Science? Bad Practices, Good Practices ConclusionFor more information• D.S. Johnson, “A Theoretician’s Guide to the ExperimentalAnalysis of Algorithms” - http://goo.gl/WxSci• T. Bartz-Beielstein, “Beyond Particular Problem Instances:How to Create Meaningful and Generalizable Results” -http://goo.gl/j7wM1• Science Blogs - Weekly Weinersmith, Bad Astronomer,Neil DeGrasse, etc

Introduction What is Science? Bad Practices, Good Practices ConclusionAcknowledgements• Based on the Experiment Design Lectures by FelipeCampelo, Federal University of Minas Gerais• Portal images copyright of Valve Software;• PhDComics images copyright of Jorge Cham;• “Harry Potter Game” taken from http://hpmor.com

Introduction What is Science? Bad Practices, Good Practices ConclusionThanks for listening!For more information and questions:caranha@cs.tsukuba.ac.jp