A Brief History of the Development of Artificial Neural Networks Prof. Bernard Widrow Department of Electrical Engineering Stanford University Baidu July 18, 2018 Prof. Widrow @ Berkeley A Brief History of Neural Networks 1/54
Adaptive linear neuron (Adaline) LMS algorithm W k+1 = W k + 2με k X k ε k = d k X k T W k Figure 1: Adaline. An adaptive linear neuron Prof. Widrow @ Berkeley A Brief History of Neural Networks 2/54
Knobby Adaline, 1959 Prof. Widrow @ Berkeley A Brief History of Neural Networks 3/54
Prof. Widrow @ Berkeley A Brief History of Neural Networks 4/54
An adaptive linear neuron(adaline) Prof. Widrow @ Berkeley A Brief History of Neural Networks 5/54
A Two-Input Adaline Prof. Widrow @ Berkeley A Brief History of Neural Networks 6/54
Separating line in pattern space Prof. Widrow @ Berkeley A Brief History of Neural Networks 7/54
Adaline Capacity Prof. Widrow @ Berkeley A Brief History of Neural Networks 8/54
Adaline with polynomial preprocessor Prof. Widrow @ Berkeley A Brief History of Neural Networks 9/54
An elliptical separating boundary for the Exclusive NOR Function Prof. Widrow @ Berkeley A Brief History of Neural Networks 10 / 54
A two-adaline form of Madaline I Prof. Widrow @ Berkeley A Brief History of Neural Networks 11 / 54
Separating lines for the two-element Madaline Prof. Widrow @ Berkeley A Brief History of Neural Networks 12 / 54
A neuronal implementation of AND, OR, and MAJ logic function Prof. Widrow @ Berkeley A Brief History of Neural Networks 13 / 54
A three-layer adaptive neural network Prof. Widrow @ Berkeley A Brief History of Neural Networks 14 / 54
Principal of Minimal Disturbance Prof. Widrow @ Berkeley A Brief History of Neural Networks 15 / 54
Error-Correction Algorithms for the Single Element Prof. Widrow @ Berkeley A Brief History of Neural Networks 16 / 54
α-lms Algorithm Prof. Widrow @ Berkeley A Brief History of Neural Networks 17 / 54
Weight correction by the LMS rule Prof. Widrow @ Berkeley A Brief History of Neural Networks 18 / 54
Rosenblatt s Perceptron Prof. Widrow @ Berkeley A Brief History of Neural Networks 19 / 54
Adaptive Threshold Element in the Perceptron Prof. Widrow @ Berkeley A Brief History of Neural Networks 20 / 54
Perceptron Rule Prof. Widrow @ Berkeley A Brief History of Neural Networks 21 / 54
May s Rule Prof. Widrow @ Berkeley A Brief History of Neural Networks 22 / 54
May s Increment Adaptation Rule Prof. Widrow @ Berkeley A Brief History of Neural Networks 23 / 54
May s Modified Relaxation Rule Prof. Widrow @ Berkeley A Brief History of Neural Networks 24 / 54
Error-Correction Algorithms for Multi-Element Networks Prof. Widrow @ Berkeley A Brief History of Neural Networks 25 / 54
A five-adaline example of the MR I Architecture Prof. Widrow @ Berkeley A Brief History of Neural Networks 26 / 54
MR II of B. Widrow and R. Winter Prof. Widrow @ Berkeley A Brief History of Neural Networks 27 / 54
Steepest-Descent Algorithms for the Single Element Prof. Widrow @ Berkeley A Brief History of Neural Networks 28 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 29 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 30 / 54
Conventional LMS Prof. Widrow @ Berkeley A Brief History of Neural Networks 31 / 54
Implementation of Conventional LMS Prof. Widrow @ Berkeley A Brief History of Neural Networks 32 / 54
Sigmoid LMS(Back-Prop) Prof. Widrow @ Berkeley A Brief History of Neural Networks 33 / 54
Implementation of Sigmoid LMS(Back-Prop) Prof. Widrow @ Berkeley A Brief History of Neural Networks 34 / 54
Sigmoid LMS(MR III) Prof. Widrow @ Berkeley A Brief History of Neural Networks 35 / 54
Implementation of Sigmoid LMS-MRIII Prof. Widrow @ Berkeley A Brief History of Neural Networks 36 / 54
Error Surfaces for the Single Element Prof. Widrow @ Berkeley A Brief History of Neural Networks 37 / 54
Linear MSE, Sigmoid MSE, and Signum MSE Prof. Widrow @ Berkeley A Brief History of Neural Networks 38 / 54
Linear Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 39 / 54
Sigmoid Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 40 / 54
Signum Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 41 / 54
Linear Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 42 / 54
Sigmoid Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 43 / 54
Signum Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 44 / 54
Steepest-Descent Algorithms for Multi-Element Networks Prof. Widrow @ Berkeley A Brief History of Neural Networks 45 / 54
Backpropagation Network Prof. Widrow @ Berkeley A Brief History of Neural Networks 46 / 54
Detail of Backpropagation Node Prof. Widrow @ Berkeley A Brief History of Neural Networks 47 / 54
MRIII of David Andes Prof. Widrow @ Berkeley A Brief History of Neural Networks 48 / 54
Prof. Widrow @ Berkeley A Brief History of Neural Networks 49 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 50 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 51 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 52 / 54
Mean Square Error Surface Prof. Widrow @ Berkeley A Brief History of Neural Networks 53 / 54
Learning Rules Prof. Widrow @ Berkeley A Brief History of Neural Networks 54 / 54