Neural Networks: Advanced

Dokuz Eylül University Computer Engineering Department

24 October 2001 – □ What is Neural Network?^{1}

• Neurons and Synapses.

• Weights, Transfer-function and Threshold.

• Layered and Fully-connected.

□ What for?

□ How can we make them work?

— Learning: specification of its weights.

• Perceptron Learning.

• Backpropagation Algorithm.

• Evolutionary Computation.

1 For more in details, see “Neural Network - Basics” in this home-page.

□ Fully-connected Neural Networks.

• Toy Model of Three Neurons:

• State Transition

s_{i} (t + 1) = sgn ( ∑_{j≠i}^{N} w_{ij} · s_{j}(t) ) .

• Example 1

An example of state transition in the above toy example:

s_{1} (1) = sgn ( w_{11} · s_{1}(0) + w_{12} · s_{2}(0) + w_{13} · s_{3}(0) )
= sgn ( 0.14 · 1 + (−0.34) · 1 + (−0.52) · 1 )
= −1

• Example 2

• Example 3 □ Associative Memory as a Dynamical System

□ A category of trajectories □ What is Associative Memory

• N neurons that are fully connected store N-bit of p patterns simultanously as fixed point attractors.^{2}

• Convergence to the initial state of the noisy stored pattern implys an error correcting capability.

• How can we determine weights.

– Hebbian Weights

w_{ij} = \frac{1}{N} ∑_{ν=1}^{p} ξ_{i}^{ν}ξ_{j}^{ν} (i ≠ j), w_{ii} = 0.

In the above toy example:

( \begin{matrix}0 & 0.11 & 0.11 \\ 0.11 & 0 & 0.11 \\ 0.11 & 0.11 & 0 \end{matrix} )

An interpretation of Hebb’s rule:

∗ A strong weight affects neurons at both ends to be same states.

– Many other solutions of a weight configuration to store a same set of patterns exist.

∗ Each can recall from · partial information; · incorrect information

∗ but has a different · basin of attraction; · storage capacity.

2 Here we restrict patterns to bipolar patterns.