%%\documentstyle[epsf]{llncs}
\documentstyle[eclepsf]{llncs}

\begin{document}

\title{Random Perturbations to Hebbian Synapses of Associative Memory using a Genetic Algorithm}

\author{Akira Imada\inst{1} and Keijiro Araki\inst{2}}

\institute{
        Graduate School of Information Science\\  
        Nara Institute of Science and Technology\\
        8916-5 Takayama, Ikoma, Nara, 630-01 Japan\\
        akira-i@is.aist-nara.ac.jp\\
        \hspace*{1in}\\
\and
        Department of Computer Science and Computer Engineering\\
        Graduate School of Information Science and Electrical Engineering\\
        Kyusyu University\\
        6-1 Kasuga-koen, Kasuga, Fukuoka, 816 Japan\\
        araki@c.csce.kyusyu-u.ac.jp}

\maketitle

\begin{abstract}
We apply evolutionary algorithms to Hopfield model of associative memory.
Previously we reported that a genetic algorithm using {\it ternary} chromosomes 
evolves the Hebb-rule associative memory to enhance its storage capacity by pruning some connections.
This paper describes a genetic algorithm using {\it real-encoded} chromosomes which successfully
evolves over-loaded Hebbian synaptic weights to function as an associative memory.
The goal of this study is to shed new light on the analysis of the Hopfield model,
which also enables us to use the model as more challenging test suite for evolutionary computations.
\end{abstract}

\section{Introduction}
\noindent
In the field of evolutionary computations, 
a lot of complicated functions have been proposed as test suites
for comparing and evaluating the effectiveness of various evolutionary computations~\cite{dejo,ackl,deb,muhl,scha,whit-2,mahf,schl}.
However as M\"{u}lenbein~\cite{muhl-2}  pointed out:
\begin{itemize}
\item[]{\it But are such problems typical applications? We have not encountered\\ such a problem.}
\end{itemize}
these functions are far from real problems. We know the number of their local optima, values, locations etc.
This makes us think of using the Hopfield model of associative memory as a test function of evolutionary computations.
The model per se is very simple, and have favorable properties as a test function of real-valued parameter optimization problems.
Firstly, as we will describe later in detail, 
for a fixed set of patterns to be stored, 
there exist almost infinite number of combinations of synaptic weights
which give the network a function of associative memory (multi modality).
Secondly, the model has a global optimum in the sense of Gardner~\cite{gard}.
Finally, there is a trade off between the storage capacity and error correcting capability (multi-objectivity).
However, we have no knowledge such as the exact distribution of these local optima.
In fact, many problems concerning the model are still open, 
despite a lot of  discussions have been made since Hopfield proposed the model.
For example, Sompolinsky~\cite{somp} studied an effect of small perturbations of Hebbian synaptic weights.
Namely, he modified the weights by
\begin{displaymath}
 w_{ij} = J_{ij} + \eta_{ij},
\end{displaymath}
where $J_{ij}$'s are Hebbian synaptic weights and $\eta_{ij}$'s are small normal random variables.
His analysis was only under the condition that the number of stored pattern was less than the storage capacity, 
and modification were made {\it randomly}.
What Sompolinsky showed is the robustness of the Hopfield model against small synaptic noises.
But what will happen if we make the experiment for more patterns than the capacity?
In this paper, we experiment it with patterns more than the saturation level,
and we modify synaptic weights {\it adaptively} using a genetic algorithm.
The genetic  algorithm uses a continuous representation instead of traditional binary representation.
We believe the study shed new light to the analysis of the model, as well as give a more challenging test suite to evolutionary computations.\\

\noindent
The remainder of this paper is organized as follows.
The other evolutionary approaches to the Hopfield model will be described  in the following section.
Since we only modify synaptic weights strictly maintaining the framework of Hopfield model, 
the model is described more fully in the third section.
Our genetic algorithm is not the standard one and these are addressed in the fourth section.
After presenting experimental results and discussion, the final section describe the concluding remarks.

\section{Related Works}
Since the late 1980's, genetic algorithms have been used extensively to search the
optimal set of synaptic weights of neural networks and/or their optimal architectures (see \cite{whit}, \cite{yao}, and references quoted therein).
However all of these researches were for layered neural networks, and applications of genetic algorithms to Hopfield networks remain few so far.\\

\noindent
Previously, we successfully evolved both random weight matrices and over-loaded Hebbian weight matrices to 
function as associative memories, using ternary chromosomes~\cite{imad-9,imad-1}. 
The chromosomes of those implementations were made up mostly 1 but a few components were 0 or -1.
Optimization was made both by pruning some synaptic connections and by balancing the number of excitatory/inhibitory synapses
with these 0 and -1, respectively.
That is, the modified synaptic weights were obtained as follows
\begin{displaymath}
 w_{ij} = c_{ij} \times J_{ij},
\end{displaymath}
or
\begin{displaymath}
 w_{ij} = c_{ij} \times R_{ij}
%%\hspace{.08in}  c_{ij} \in \{1, 0, -1\},
\end{displaymath}
where $c_{ij}$'s are components of the chromosome and $R_{ij}$'s are Random synaptic weights.\\

\noindent
The simulations were made with 49 neurons as in this paper.
The random matrix evolved to store eventually 7 patterns as fixed points.
And starting with over-loaded Hebbian matrix which learned 16 patterns 
it was evolved to stabilize all these 16 patterns as fixed points.
Note that this size of Hebbian matrix would store at most 8 patterns.

\section{Hoppfield Model of Associative Memory}
Associative memory is a dynamical system which has a number of stable states with a domain of attraction around them \cite{koml}.
Applying a vector 
\( \mbox{\boldmath $x$}_{0} \) 
as the initial state of the dynamical system results in the state trajectory
\( \{ \mbox{\boldmath $x$}_{t} \}_{t \ge 0} \).
If we start the system with 
\( \mbox{\boldmath $x$}_{0}=\mbox{\boldmath $u$} + \mbox{\boldmath $\delta u$} \) 
in the vicinity of some fixed point \( \mbox{\boldmath $u$} \), 
and the state trajectory converges to \( \mbox{\boldmath $u$} \), i.e., 
\( \lim_{t \to \infty} \mbox{\boldmath $x$}_{t} = \mbox{\boldmath $u$} \),
then we can regard the system as an associative memory.\\

\noindent
In 1982, Hopfield proposed his fully connected neural network model of associative memory \cite{hopf}.
The model consists of $N$ neurons and $N^2$ synapses.
Each neuron can be in one of two states $\pm 1$, and $p$ bipolar patterns 
%%\begin{displaymath}
$
{\xi}^{\nu} = ({\xi}^{\nu}_{1},\cdots,{\xi}^{\nu}_{N}) \hspace{.1in}({\nu} = 1, \cdots, p)
$
%%\end{displaymath}
are memorized as equilibrium states.
He employed a discrete-time, asynchronous update scheme.
Namely, at most one neuron updates its state at a time, according to 
%%the sign of the weighted sum of of the states of all other neurons, i.e.,
\begin{displaymath}
%%$
s_{i}\hspace{.02in} (t+1) = f \hspace{.02in} \left( \hspace{.04in} \sum_{j \ne i}^N { w_{ij} s_{j}^{\mu}(t)} \hspace{.04in} \right),
%%$
\end{displaymath}
where $s_{i}(t)$ is a state of $i$-th neuron at time $t$, $w_{ij}$ is a synaptic weight from neuron $j$ to neuron $i$, 
and $f(z) = 1$ if $z \ge 0$ and $-1$ otherwise. \\

\noindent
The behaviors of the collective states of the individual neurons are characterized by the synaptic weights.
%%the $N^{2}$ synaptic weights $w_{ij}$.
When these synaptic weights are determined appropriately, networks store some number of patterns as fixed points.
Hopfield specified $w_{ij}$'s by the Hebbian rule \cite{hebb}, i.e.,
\begin{displaymath}
%%$
w_{ij} = \frac{1}{N} \sum _{ \nu = 1}^{p} {{ \xi }_{i}^{ \nu }{ \xi }_{j}^{ \nu }}\hspace{.04in}(i \ne j), \hspace{.02in} w_{ii} = 0.
%%$
\end{displaymath}
Then giving one of the memorized patterns, possibly including a few errors, to the network as an initial state
will result in the stable state after certain time steps.
Hopfield suggested by using computer simulation that the maximum number $p$ of the patterns to be stored in a network with $N$ neurons is $0.15N$,
if a small error in recalling is allowed.
Later, this was theoretically calculated by Amit et al.~\cite{ags} by using {\it replica} method. 
They showed that the storage capacity is $ p=0.14N $.
%%Amit, Gutfreund \& Sompolinsky\cite{amit}
In 1987, McEliece et al.~\cite{mcel} proved that when
$
p < N / 4 \ln N
$
holds, the Hopfield's model will recall the memory without error.\\

\noindent
The storage capacity depends strongly on how synaptic weights are specified. 
The specification of the synaptic weights is conventionally referred to as {\it learning}.
Other learning schemes instead of Hebbian rule have been proposed for their increased storage capacity. 
The {\it pseud-inverse matrix method} by Kohonen et al.~\cite{koho}
and {\it spectral algorithm} by Pancha et al.~\cite{panc} 
are the examples, and the capacity is enlarged to $ p=0.5N $ and $p=N$, respectively.
Both are extensions of the Hebbian learning.\\

\noindent
Then, what is the ultimate capacity when only the learning scheme is explored within the Hopfield's framework?
Generally the storage capacity is traded off against basin size.
Gardner\cite{gard} studied the optimal storage capacity as a function of size of basin of attraction.
She showed that as basin size tends to zero, the ultimate capacity will be $p=2N$, proposing an algorithm to obtain the weight matrix.

\section{GA Implementation}
\noindent
In this simulation, a set of random bipolar patterns ${\xi}^{\nu}$ (${\nu} = 1, 2, \cdots, p$) are produced before each run.
The goal of the genetic algorithm is to find a weight matrix which stores all these patterns as fixed points.
A weight matrix $\{W_{ij}\}$ is also pre-determined as an ancestor. This matrix remains fixed during evolution.
Then a population of chromosomes are initialized at the beginning of a run. 
Each chromosome is a $N^2$-dimensional real-valued vector. Its component is called as allele. We denote the chromosome as
\begin{displaymath}
(\eta_{11}, \eta_{12}, \cdots, \eta_{1N}, \eta_{21}, \cdots \cdots \eta_{NN})
\end{displaymath}
In each generation, a population of chromosomes make ancestor's copies by adding each allele to its corresponding component of the ancestor matrix.
Namely, 
\begin{displaymath}
w^{k}_{ij} = W_{ij} + \eta^{k}_{ij},
\end{displaymath}
where $w^{k}_{ij}$ is $ij$-component of the copied matrix, $\eta^{k}_{ij}$ is $(iN+j)$-th allele of the chromosome, 
and the superscript $k$ denotes the individual number in the population.  
The chromosomes are modified through crossover and mutation operation.
The fitness values of the corresponding phenotype are evaluated.
According to the fitness values, individuals of the next generation are selected,
using a ($\mu + \lambda$)-strategy in ES terminology.
The cycle of reconstructing the new population with better individuals and restarting the search is repeated until a perfect
solution is found or a maximum number of generation has been reached.
Specific details of the operations are  as follows.

\begin{description}
\item[]{\bf Initialization:}
In the first generation, all chromosomes are initialized 
so that each allele value is Gaussian random variables with mean 0 and standard deviation $\sigma$.
\item[] {\bf Fitness evaluation:}
When one of memorized patterns ${\xi}^{\nu}$ is given to the network as an initial state, 
the state of neurons varies from time to time afterwards (unless ${\xi}^{\nu}$ is a fixed point).
In order for the network to function as associative memory, the instantaneous network state must be similar to the input pattern. 
The similarity is defined by,
\begin{displaymath}
m^{\nu}(t) = \frac{1}{N} \sum_{i=1}^{N} { {\xi}_{i}^{\nu} s^{\nu}_{i}(t) },
\end{displaymath}
where $s_{i}^{\nu}(t)$ is the state of neuron $i$ at time $t$. 
%%Clearly it is related to Hamming distance between the two states.
This $m^{\nu}(t)$ is referred to as {\it overlap}.
The quality of retrieval of memorized patterns are represented by $\left \langle m^{\mu} \right \rangle$, 
a temporal average of $m^{\nu}(t)$ over certain time interval $t_0$.
We evaluate the fitness value of each network by further averaging the $\left \langle m^{\mu} \right \rangle$ over all memorized patterns. 
Namely, the fitness $f$ is
\begin{displaymath}
f = \frac {1}{t_{0} \cdot p}
\sum_{t=1}^{t_{0}}{\sum _{\nu=1}^{p}{m^{\nu}(t)} }.
\end{displaymath}
In this paper, $t_{0}$ is set to $2N$, twice the number of neurons.
Note that the fitness 1 implies all the $p$ patterns are stored as fixed points, while fitness less than 1 includes many other cases.
%%The attractor distributions are parametrized by these $p$ overlaps $m^{\nu}$.

\item[] {\bf Selection:}
We use the truncation-selection~\cite{muhl-2}, i.e., two parent chromosomes are chosen {\it randomly} from the best $T$\% of the population.
\item[] {\bf Recombination:}
Discrete Recombination \cite{muhl-2} which is similar to {\it uniform crossover} \cite{sysw} in binary chromosomes, 
operates on alleles of the selected parents' chromosomes, i.e., 
two parents $(u_1, \cdots, u_n)$ and $(v_1, \cdots, v_n)$ produce an offspring $(w_1, \cdots, w_n)$
such that $w_i$ is either $u_i$ or $v_i$ with equal probability.

\item[] {\bf Mutation:}
Mutation is made by randomly picking up an allele with probability $p_m$, 
and replacing it with uniformly random number selected from $[-\delta, \delta]$.\\
\end{description}


\section{Experiment}
\noindent
The parameters we used in the genetic algorithm are as follows.
The population number is 256. This is just because of our computer resources.
The standard deviation $\sigma$ in initializing allele values in the first generation is set to 0.01.
The mutation probability $p_m$ and the mutation range $\delta$ are set to 0.05 and 0.1, respectively.
In truncation selection T is chosen to be 40\%.  
Searching procedure is iterated until 12000 generation unless perfect solution is not found.
These were determined mainly on the basis of trial and error.
%%All simulations were carried out on networks with 49 neurons.

\section{Results and Discussion}
\begin{figure}[hbtp]
 \begin{center}
   \epsfile{file=fitness.eps,width=7cm,height=7cm}
   \caption{Fitness vs. generation}
   \label{fitness}
 \end{center}
\end{figure}
In this experiment, we start the genetic algorithm with a Hebbian weight matrix which learned a set of patterns more than capacity.
The networks of this type were extensively investigated by Amit et al.~\cite{ags,ags-2}.
As the number of patterns increases, the correlation of these patterns can not be neglected.
when the number exceeds the capacity, some memory states are shifted to their neighborhood, 
and spurious attractors around memory states merge into spin-glas attractors.\\

\noindent
Let us take an example. We made an experiment for 17 patterns to be memorized on a network with 49 neurons.
Since the storage capacity of  the Hebbian synaptic weights of this size would be around 8 patterns,
these 17 patterns are a little more than twice of the capacity.
Each input of memorized patterns results in the convergence to the stable attractors though, 
most of them are not memorized states.
These attractors can be studied by the temporal evolution of 
Hamming distance of instantaneous network state from the corresponding memorized pattern.
We observed that the Hamming distances of the 17 stable states from their corresponding input were
0, 0, 0, 0, 1, 1, 1, 6, 7, 7, 8, 10, 10, 14, 15, 18 and 19.
The four are fixed-point attractors and the rest are distributed from near-fixed-point attractors to spin-glass-attractors.\\

\noindent
We use this weight matrix as an ancestor matrix for our genetic algorithm as described above.
Surprisingly we obtained a evolved matrix which store all these 17 patterns as fixed points perfectly after 3273 generations.
We show the fitness evolution in Figure~\ref{fitness}.\\

\noindent
While we obtained the perfect solutions more easily for less than 16 patterns (not in the Figure), 
we have not observed the same success for 18 patterns among 30 runs with different random number seed. \\

\noindent
As Hertz et al.~\cite{herz} and Parisi~\cite{pari} suggested in their analysis using Ising model,
we conjectured that the above success is due to the destabilization of the spin-glass attractors 
by the asymmetry of synaptic weights introduced by the genetic algorithm.\\

\noindent
\begin{figure}[hbtp]
 \begin{center}
   \epsfile{file=symmetry.eps,width=7cm,height=7cm}
   \caption{Degree of symmetry of the matrix which evolved to perfect solution.}
   \label{symmetry}
 \end{center}
\end{figure}

\noindent
To see this, we investigated the time evolution of degree-of-symmetry of weight matrices.
The degree-of-symmetry is defined by
\begin{displaymath}
\sum_{i=1}^{N} \sum_{j=1}^{N} {w_{ij} w_{ji}} \cdot \left( \sum_{i=1}^{N} \sum_{j=1}^{N} {w_{ij}^{2}} \right)^{-1},
\end{displaymath}
following after Krauth et al.~\cite{krau}.
We show the results of both experiments in Figure~\ref{symmetry}. 
At the beginning the Hebbian matrix, however over-loaded it may be, is totally symmetric.
As evolution proceeds the degree of symmetry decreases and 
asymptotically approaches to the value around 0.8 and retrieval states emerge.
Our conjecture is that the asymmetry destabilized the many spin-glass attractors which existed at the beginning.\\

\noindent
However the role of asymmetry of the synaptic weights is still an open question.

\noindent
To say a network an associative memory system, the tolerance for certain amount of noises should be shown.
That is, each memorized pattern must have a certain domain of basin of attraction.\\

\noindent
We observed the basin size of the network of the above experiment as follows.
One of the memorized patterns are chosen and flipped $d$ bits at random.
The pattern is given to the network, and temporal average of {\it overlaps} of the network state visited by the dynamics is calculated.
This is repeated 800 times for each incremented $d$. 
In Figure~\ref{basin}, the averaged overlap over repetition are plotted against noisy-bit $d$, 
together with the same experiment with Gardner's weight matrix for the same set of memorized patterns.
Although the size of basin of attraction is much smaller than that of Gardner's optimized matrix, it still shows some tolerance for noises.

\begin{figure}[hbtp]
 \begin{center}
   \epsfile{file=basin.eps,width=7cm,height=7cm}
   \caption{Similarity of updated states with noisy input}
   \label{basin}
 \end{center}
\end{figure}

\section{Conclusion}
We have described an application of genetic algorithms to Hopfield model of associative memory.
We give small perturbations to Hebbian synaptic weights.
This is essentially the same as Sompolinsky's model, 
with difference being that we use an over-saturated network and choose synapses {\it adaptively},
while Sompolinsky used an unsaturated network and chose synapses {\it randomly}.
The simulations in this paper were made with 49 neurons.
This size of Hebb-rule associative memory would store at most 8 patterns as fixed points.
We succeeded in evolving a network which learned 17 patterns by Hebbian rule to store eventually all these patterns as fixed points.\\

\noindent
Although our investigations have not been conclusive so far,
we believe this study shed new light on the analysis of Hopfield model from new aspect,
and give new test suite for evolutionary computations.

\section*{Acknowledgments}
\noindent
We thank Peter Davis at Advanced Telecommunication Research Institute (ATR)
for providing us great insight into the dynamics of Hopfield neural networks.

\begin{thebibliography}{5}

\bibitem{dejo} K. DeJong (1975)
{\it An Analysis of the Behavior of a Class of Genetic Adaptive Systems.}
Ph.D thesis, University of Michigan. 

\bibitem{ackl}D. Ackley (1987)
{\it A Connectionist Machine for Genetic Hillclimbing.}
Boston: Kluwer. 

\bibitem{deb} K. Deb (1989)
{\it Genetic Algorithms in Multimodal Function Optimization.}
Master's thesis.  The University of Alabama.

\bibitem{muhl}H. M\"{u}hlenbein, M. Schomish,  and J. Born (1991)
{\em The Parallel Genetic Algorithm as Function Optimizer.}
Parallel Computing {\bf 17} pp619--632. 

\bibitem{scha}J. D. Schaffer, B. A. Caruana, L. J. Eshlman, and R. Das (1991)
{\em A Study of Control Parameters Affecting Online Performance of Genetic Algorithms for Function Optimization.}
Proceedings of the 3rd International Conference on Genetic Algorithms. pp51--60.

\bibitem{whit-2}D. Whitley, K. Mathias, S. Rana, and J. Dzubera (1995)
{\em Building Better Test Functions.}
Proceedings of the 6th International Conference on Genetic Algorithms. pp239-247.

\bibitem{mahf}S. M. Mahfoud (1995)
{\em A. Comparison of Parallel and Sequential Niching Methods.}
Proceedings of the 6th International Conference on Genetic Algorithms. pp239-247.

\bibitem{schl}D. Schlierkamp-Voosen, and H. M\"{u}hlenbein (1996)
{\em Adaptation of Population Sizes by Competing Subpopulation.}
Proceedings of IEEE International Conference on Evolutionary Computation. pp330-335.

\bibitem{muhl-2}H. M\"{u}hlenbein, and D. Schlierkamp-Voosen (1995)
{\em Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization.}
Evolutionary Computation {\bf 1} (1996)  pp25--49.

\bibitem{gard} E. Gardner (1988)
{\em The Phase Space of Interactions in Neural Network Models.}
J. Phys A: Math Gen. {\bf 21}. pp257--270.

\bibitem{somp}H. Sompolinsky (1986)
{\em Neural Network with Non-linear Synapses and Static Noise.}
Phys. Rev. {\bf A34}  pp2571-2574.

\bibitem{whit}J. D. Schaffer, D. Whitley, and L. J. Eshelman (1992)
{\em Combinations of Genetic Algorithms and Neural Networks: A Survey of the State of the Art.}
Proceedings of the Workshop on Combinations of Genetic Algorithms and Neural Networks. pp1--37.

\bibitem{yao}X. Yao (1993)
{\em A Review of Evolutionary Artificial Neural Networks.}
International Journal of Intelligent Systems {\bf 8}. pp539--567.

\bibitem{imad-9}A. Imada, and K. Araki (1997)
{\em Evolved Asymmetry and Dilution of Random Synaptic Weights in Hopfield Network Turn a Spin-glass Phase into Associative Memory.}
The 2nd International Conference on Computational Intelligence and Neuroscience (to appear)

\bibitem{imad-1}A. Imada, and K. Araki (1995)
{ \em Genetic Algorithm Enlarges the Capacity of Associative Memory.}
Proceedings of the 6th International Conference on Genetic Algorithms. pp413--420.

\bibitem{koml}J.~Koml\'{o}s, and R. Paturi (1988)
{\em Convergence Results in an Associative Memory Model.} 
Neural Networks {\bf 1}. pp239--250.

\bibitem{hopf}J.~J.~Hopfield (1982) 
{\em Neural Networks and Physical Systems with Emergent Collective Computational Abilities.}
Proceedings of the National Academy of Sciences, USA. {\bf 79}.  pp2554--2558.

\bibitem{hebb}D.~O.~Hebb (1949)
{\it The Organization of Behavior.} Wiley.

\bibitem{ags}D. J. Amit, H.~Gutfreund, and H.~Sompolinsky (1985)
{\em Statistical Mechanics of Neural Networks near Saturation}
Annals. of Physics 173, pp30-67.

\bibitem{mcel} R. J. McEliced, E. C. Posner, E.~R.~Rodemick, and S.~S.~Venkatesh (1987)
{\em The Capacity of the Hopfield Associative Memory.}
IEEE Trans. Information Theory {\bf IT-33}. pp461-482.

\bibitem{koho} T. Kohonen, and M. Ruohonen (1973)
{\em  Representation of Associated Data by Matrix Operators.}
IEEE Trans. Computers {\bf C-22(7)}.  pp701--702.

\bibitem{panc}G. Pancha, S. S. VenKatesh (1993)
{\em Feature and Memory-Selective Error correction in Neural Associative Memory.}
in M.~H.~Hassoun eds. Associative Neural Memories: Theory and Implementation.
Oxford University Press. pp225-238.

\bibitem{sysw}G. Syswerda (1989)
{\em Uniform Crossover in Genetic Algorithms.}
Proceedings of the 3rd International Conference on Genetic Algorithms, pp2--9.

\bibitem{ags-2}D. J. Amit, H.~Gutfreund, and H.~Sompolinsky (1985)
{\em Storing Infinite Number of Patterns in a Spin-glass Model of Neural Networks.}
Physical Review Letters {\bf 55}. pp1530-1533.

\bibitem{herz} J. A. Hertz, G. Grinstein, and S. A. Solla (1987)
{\em Irreversible Spin Glasses an Neural Networks.}
in L. N. van Hemmen and I. Morgenstern eds. Heidelberg Colloquium on Glassy Dynamics. Lecture Notes in Physics No.275 Springer-Verlag, pp538--546.

\bibitem{pari} G. Parisi (1986)
{\em Asymmetric Neural Networks and the Process of Learning.} J. Phys. {\bf A19} ppL675--L680.

\bibitem{krau}W. Krauth, J.-P. Nadal, and M.Mezard (1988)
{\em The Roles of Stability and Symmetry in the Dynamics of Neural Networks.}
J. Phys. A: Math. Gen. {\bf 21} pp2995--3011.

\end{thebibliography}

\end{document}
.
