A Brief History of the Development of Artificial Neural Networks

Size: px

Start display at page:

Download "A Brief History of the Development of Artificial Neural Networks"

Valentine Burke
5 years ago
Views:

1 A Brief History of the Development of Artificial Neural Networks Prof. Bernard Widrow Department of Electrical Engineering Stanford University Baidu July 18, 2018 Prof. Berkeley A Brief History of Neural Networks 1/54

2 Adaptive linear neuron (Adaline) LMS algorithm W k+1 = W k + 2με k X k ε k = d k X k T W k Figure 1: Adaline. An adaptive linear neuron Prof. Berkeley A Brief History of Neural Networks 2/54

3 Knobby Adaline, 1959 Prof. Berkeley A Brief History of Neural Networks 3/54

4 Prof. Berkeley A Brief History of Neural Networks 4/54

5 An adaptive linear neuron(adaline) Prof. Berkeley A Brief History of Neural Networks 5/54

6 A Two-Input Adaline Prof. Berkeley A Brief History of Neural Networks 6/54

7 Separating line in pattern space Prof. Berkeley A Brief History of Neural Networks 7/54

8 Adaline Capacity Prof. Berkeley A Brief History of Neural Networks 8/54

9 Adaline with polynomial preprocessor Prof. Berkeley A Brief History of Neural Networks 9/54

10 An elliptical separating boundary for the Exclusive NOR Function Prof. Berkeley A Brief History of Neural Networks 10 / 54

11 A two-adaline form of Madaline I Prof. Berkeley A Brief History of Neural Networks 11 / 54

12 Separating lines for the two-element Madaline Prof. Berkeley A Brief History of Neural Networks 12 / 54

13 A neuronal implementation of AND, OR, and MAJ logic function Prof. Berkeley A Brief History of Neural Networks 13 / 54

14 A three-layer adaptive neural network Prof. Berkeley A Brief History of Neural Networks 14 / 54

15 Principal of Minimal Disturbance Prof. Berkeley A Brief History of Neural Networks 15 / 54

16 Error-Correction Algorithms for the Single Element Prof. Berkeley A Brief History of Neural Networks 16 / 54

17 α-lms Algorithm Prof. Berkeley A Brief History of Neural Networks 17 / 54

18 Weight correction by the LMS rule Prof. Berkeley A Brief History of Neural Networks 18 / 54

19 Rosenblatt s Perceptron Prof. Berkeley A Brief History of Neural Networks 19 / 54

20 Adaptive Threshold Element in the Perceptron Prof. Berkeley A Brief History of Neural Networks 20 / 54

21 Perceptron Rule Prof. Berkeley A Brief History of Neural Networks 21 / 54

22 May s Rule Prof. Berkeley A Brief History of Neural Networks 22 / 54

23 May s Increment Adaptation Rule Prof. Berkeley A Brief History of Neural Networks 23 / 54

24 May s Modified Relaxation Rule Prof. Berkeley A Brief History of Neural Networks 24 / 54

25 Error-Correction Algorithms for Multi-Element Networks Prof. Berkeley A Brief History of Neural Networks 25 / 54

26 A five-adaline example of the MR I Architecture Prof. Berkeley A Brief History of Neural Networks 26 / 54

27 MR II of B. Widrow and R. Winter Prof. Berkeley A Brief History of Neural Networks 27 / 54

28 Steepest-Descent Algorithms for the Single Element Prof. Berkeley A Brief History of Neural Networks 28 / 54

29 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 29 / 54

30 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 30 / 54

31 Conventional LMS Prof. Berkeley A Brief History of Neural Networks 31 / 54

32 Implementation of Conventional LMS Prof. Berkeley A Brief History of Neural Networks 32 / 54

33 Sigmoid LMS(Back-Prop) Prof. Berkeley A Brief History of Neural Networks 33 / 54

34 Implementation of Sigmoid LMS(Back-Prop) Prof. Berkeley A Brief History of Neural Networks 34 / 54

35 Sigmoid LMS(MR III) Prof. Berkeley A Brief History of Neural Networks 35 / 54

36 Implementation of Sigmoid LMS-MRIII Prof. Berkeley A Brief History of Neural Networks 36 / 54

37 Error Surfaces for the Single Element Prof. Berkeley A Brief History of Neural Networks 37 / 54

38 Linear MSE, Sigmoid MSE, and Signum MSE Prof. Berkeley A Brief History of Neural Networks 38 / 54

39 Linear Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 39 / 54

40 Sigmoid Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 40 / 54

41 Signum Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 41 / 54

42 Linear Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 42 / 54

43 Sigmoid Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 43 / 54

44 Signum Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 44 / 54

45 Steepest-Descent Algorithms for Multi-Element Networks Prof. Berkeley A Brief History of Neural Networks 45 / 54

46 Backpropagation Network Prof. Berkeley A Brief History of Neural Networks 46 / 54

47 Detail of Backpropagation Node Prof. Berkeley A Brief History of Neural Networks 47 / 54

48 MRIII of David Andes Prof. Berkeley A Brief History of Neural Networks 48 / 54

49 Prof. Berkeley A Brief History of Neural Networks 49 / 54

50 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 50 / 54

51 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 51 / 54

52 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 52 / 54

53 Mean Square Error Surface Prof. Berkeley A Brief History of Neural Networks 53 / 54

54 Learning Rules Prof. Berkeley A Brief History of Neural Networks 54 / 54

Neural Nets Using Backpropagation. Chris Marriott Ryan Shirley CJ Baker Thomas Tannahill

Neural Nets Using Backpropagation Chris Marriott Ryan Shirley CJ Baker Thomas Tannahill Agenda Review of Neural Nets and Backpropagation Backpropagation: The Math Advantages and Disadvantages of Gradient