Inductive Bias 1 - Neural Networks and Machine Learning Lab

CS 478 - Inductive Bias 1

Noise vs. ExceptionsCS 478 - Inductive Bias 2

The Hypothesis space H is the set all the possible models hwhich can be learned by the current learning algorithm– e.g. Set of possible weight settings for a perceptron Restricted hypothesis space– Can be easier to search– May avoid overfit since they are usually simpler (e.g. linear or loworder decision surface)– Often will underfit Unrestricted Hypothesis Space– Can represent any possible function and thus can fit the training setwell– Mechanisms must be used to avoid overfitCS 478 - Inductive Bias 3

Simplest accurate model: accuracy vs. complexity tradeoff.Find h ∈ H which minimizes an objective function ofthe form:F(h) = Error(h) + λ·Complexity(h)– Complexity could be number of nodes, size of tree, magnitude ofweights, order of decision surface, etc. More Training Data (vs. overtraining on same data) Stopping criteria with any constructive model (Accuracyincrease vs Statistical significance) – Noise vs. Exceptions Validation Set (next slide: requires separate test set) Will discuss other approaches laterCS 478 - Inductive Bias 4

SSEValidation SetEpochs (new h at each)Training SetThere is a different model h after each epochSelect a model in the area where the validation set accuracy flattensThe validation set comes out of training set dataStill need a separate test set to use after selecting model h to predictfuture accuracyCS 478 - Inductive Bias 5

The approach used to decide how to generalize novel casesOne common approach is Occam’s Razor – The simplest hypothesiswhich explains/fits the data is usually the bestMany other rationale biasesABC ⇒ ZAB C ⇒ ZABC ⇒ ZAB C ⇒ ZA B C ⇒ ZA BC ⇒ ?When you get the new input Ā B C. What is your output?€CS 478 - Inductive Bias 6

Inductive Bias: Any basis for choosing one generalizationover another, other than strict consistency with theobserved training instancesSometimes just called the Bias of the algorithm (don't confusewith the bias weight in a neural network).CS 478 - Inductive Bias 7

Restricted Hypothesis Space - Can just try tominimize error since hypotheses are already simple– Linear or low order threshold function– k-DNF, k-CNF, etc.– Low order polynomial Preference Bias – Prefer one hypothesis over anothereven though they have similar training accuracy– Occam’s Razor– “Smallest” DNF representation which matches well– Shallow decision tree with high information gain– Neural Network with low validation error and smallmagnitude weightsCS 478 - Inductive Bias 8

2 2n Boolean functions of n inputsx1 x2 x3 Class Possible Consistent Function Hypotheses0 0 0 10 0 1 10 1 0 10 1 1 11 0 01 0 11 1 01 1 1 ?CS 478 - Inductive Bias 9

2 2n Boolean functions of n inputsx1 x2 x3 Class Possible Consistent Function Hypotheses0 0 0 1 10 0 1 1 10 1 0 1 10 1 1 1 11 0 0 01 0 1 01 1 0 01 1 1 ? 0CS 478 - Inductive Bias 10

2 2n Boolean functions of n inputsx1 x2 x3 Class Possible Consistent Function Hypotheses0 0 0 1 1 10 0 1 1 1 10 1 0 1 1 10 1 1 1 1 11 0 0 0 01 0 1 0 01 1 0 0 01 1 1 ? 0 1CS 478 - Inductive Bias 11

2 2n Boolean functions of n inputsx1 x2 x3 Class Possible Consistent Function Hypotheses0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 11 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 11 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 11 1 1 ? 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1CS 478 - Inductive Bias 12

2 2n Boolean functions of n inputsx1 x2 x3 Class Possible Consistent Function Hypotheses0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 10 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 11 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 11 1 0 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 11 1 1 ? 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1What happens in this case if we use simplicity (Occam’s Razor) asour inductive BiasCS 478 - Inductive Bias 13

“Raster Screen” ProblemPattern Theory– Regularity in a task– CompressibilityDon’t care features and Impossible statesInteresting/Learnable Problems– What we actually deal with– Can we formally characterize them?Learning a training set vs. generalizing– A function where each output is set randomly (coin-flip)– Output class is independent of all other instances in the data setComputability vs. Learnability (Optional)CS 478 - Inductive Bias 14

Finite problems assume finite number of mappings (Finite Table)– Fixed input size arithmetic– Random memory in a RAM Learnable: Can do better than random on novel examplesCS 478 - Inductive Bias 15

Finite problems assume finite number of mappings (Finite Table)– Fixed input size arithmetic– Random memory in a RAM Learnable: Can do better than random on novel examplesFinite ProblemsAll are ComputableLearnable Problems:Those with RegularityCS 478 - Inductive Bias 16

Infinite number of mappings (Infinite Table)– Arbitrary input size arithmetic– Halting Problem (no limit on input size)– Do two arbitrary strings matchCS 478 - Inductive Bias 17

Infinite number of mappings (Infinite Table)– Arbitrary input size arithmetic– Halting Problem (no limit on input size)– Do two arbitrary strings matchInfinite ProblemsLearnable Problems:A reasonably queriedinfinite subset has regularityComputable Problems:Only those where all but a finiteset of mappings have regularityCS 478 - Inductive Bias 18

Any inductive bias chosen will have equal accuracy compared to anyother bias over all possible functions/tasks, assuming all functions areequally likely. If a bias is correct on some cases, it must be incorrecton equally many cases.Is this a problem?– Random vs. Regular– Anti-Bias? (even though regular)– The “Interesting” Problems – subset of learnable?Are all functions equally likely in the real world?CS 478 - Inductive Bias 19

Interesting Problems and BiasesAll ProblemsStructured ProblemsInteresting ProblemsInductive BiasInductive BiasInductive BiasP IInductive BiasInductive BiasCS 478 - Inductive Bias20

Inductive Bias requires some set of prior assumptionsabout the tasks being considered and the learningapproaches available Mitchell’s definition: Inductive Bias of a learner is the setof additional assumptions sufficient to justify its inductiveinferences as deductive inferences We consider standard ML algorithms/hypothesis spaces tobe different inductive biases: C4.5 (Greedy bestattributes), Backpropagation (simple to complex), etc.CS 478 - Inductive Bias 21

Not one Bias that is best on all problems Our experiments– Over 50 real world problems– Over 400 inductive biases – mostly variations on critical variablebiases vs. similarity biases Different biases were a better fit for different problems Given a data set, which Learning model (Inductive Bias)should be chosen?CS 478 - Inductive Bias 22

Defining and characterizing the set of Interesting/Learnable problemsTo what extent do current biases cover the set of interesting problemsAutomatic feature selectionAutomatic selection of Bias (before and/or during learning), includingall learning parametersDynamic Inductive Biases (in time and space)Combinations of Biases – Ensembles, Oracle LearningCS 478 - Inductive Bias 23

Can be discovered as you learn May want to learn general rules first followed by trueexceptions Can be based on ease of learning the problem Example: SoftProp – From Lazy Learning to BackpropCS 478 - Inductive Bias 24

CS 478 - Inductive Bias 25

Just aData Setorjust anexplanationof the problemAutomatedLearnerOutputsHypothesisInput FeaturesCS 478 - Inductive Bias 26

Proposing New Learning Algorithms (Inductive Biases)Theoretical issues– Defining the set of Interesting/Learnable problems– Analytical/empirical studies of differences between biasesEnsembles – Wagging, Mimicking, Oracle Learning, etc.Meta-Learning – A priori decision regarding which learning model to use– Features of the data set/application– Learning from model experienceAutomatic selection of Parameters– Constructive Algorithms – ASOCS, DMPx, etc.– Learning Parameters – Windowed momentum, Automatic improved distancefunctions (IVDM)Automatic Bias in time – SoftPropAutomatic Bias in space – Overfitting, sensitivity to complex portions of thespace: DMP, higher order featuresCS 478 - Inductive Bias 27