Lamarckian Evolution of Associative Memory

Akira Imada

Graduate School of Information Science Nara Institute of Science and Technology

8916-5 Takayama, Ikoma, Nara,

630-01 Japan akira-i@is.aist-nara.ac.jp

Keijiro Araki

Graduate School of Information Science Nara Institute of Science and Technology

8916-5 Takayama, Ikoma, Nara, 630-01 Japan araki@is.aist-nara.ac.jp

Abstract— There have been a lot of researches which apply evolutionary techniques to layered neural networks. However, their applications to Hopfield neural networks remain few so far. We are applying genetic algorithms to fully connected associative memory model of Hopfield. In an earlier paper, we reported that random weight matrices were evolved to store some number of patterns only by means of a simple genetic algorithm. In this paper we present that the storage capacity can be enlarged by incorporating Lamarckian inheritance to the genetic algorithm.

I. Introduction

Associative memory is a process in which we can store information by distributing it among neurons. One of the most appealing features is its capability to tolerate noisy and/or partial input in recalling the information, though it has a limited capacity. We are thinking of using this as a memory organ of digital organisms of our Artificial Life research in the future. However, as a preliminary step, we are now investigating basic behavior of the function of associative memory under simple evolutionary processes.

In 1982 Hopfield [?] proposed a neural network model as an associative memory system. He used Hebbian rule [?] to make the network store a set of patterns. He estimated the storage capacity to be at most 15% of the number of neurons with computer simulations. Later the capacity was verified more analytically by Amit et al. [?] with spin-glass theory, and by Amari et al. [?] with statistical neuro-dynamical method. Since then, many researchers have been successfully trying to enlarge the capacity in various ways (see [?]). For example, Yanai et al. [?] made it by introducing sparse connectivity, Morita [?] by replacing the sigmoid function with a non-monotonic transfer function, and Willshow et al. [?], Palm [?] and Meunier et al. [?] by using sparsely coded patterns instead of random patterns. In this paper, we are exploring this by means of a genetic algorithm.

Genetic algorithms are a kind of searching techniques [?], [?]. Since the late 1980’s, genetic algorithms have been extensively used to search the optimal set of connection weights of neural networks and/or their optimal architectures (for reviews of this issue, see [?], [?] and references quoted therein). At first, they were used for determining connection weights, (see e.g., [?], [?]). However, it was pointed out that the genetic algorithms as a learning method were not so effective compared to other algorithms like Back Propagation [?]. So the interests were shifted toward the exploration of neural network architectures (see e.g., [?], [?]). However, to represent architectures effectively, more sophisticated genetic algorithms had to be considered. For example, grammatical representations of the architecture of neural networks were proposed [?], [?].

All of these researches were for layered neural networks, and there are only a few for Hopfield networks [?], [?], probably because the Hopfield network has a fixed architecture. However, sparse connectivity has a possibility to enhance the original associative memory capacity of Hopfield network [?]. Therefore, genetic algorithms are also valid for Hopfield neural networks to explore their architectures as well as to determine their connection weights.

In an earlier paper, we showed that a genetic algorithm enlarges the capacity of Hebb-rule associative memory by pruning some connections [?]. We also showed that randomly generated connection matrices can learn some patterns using a genetic algorithm without any learning rules [?].

In this paper, we redesigned the genetic algorithm so that we can test the effect of Lamarckian inheritance on the evolution. In this implementation, each individual in the population learn a set of pre-determined random patterns in its life time, and the learning result affects its chromosome. And we found that it can enlarge the storage capacity.

The overall goal for this research is twofold. One is to know if we can use the genetic algorithm as a more effective learning method than those proposed so far, and the other is to clarify the mechanism of this learning on an individual level during its life time as well as the learning on a population level of the genetic algorithm.

In the following section, we provide an outline of our experiment. Then the effects of Lamarckian inheritance on the evolution are described in Section III. Finally some conclusions are given.

II. Experiment

We applied a simple genetic algorithm to fully connected associative memory model of Hopfield. To be more specific, an auto-associative, discrete-time asynchronous-update, bipolar-state network.

The connection weights are chosen randomly either from −1 or 1 at the start of the genetic algorithm, therefore the network can not store any patterns initially. The objective of the experiment is to make this network store some num-