\documentclass[10pt]{article}
\usepackage{graphicx}

\textheight 225mm
%\topmargin 5mm
\textwidth 175mm
\oddsidemargin -7mm

\newtheorem{algorithm}{Algorithm}
\newtheorem{challenge}{Challenge}

\title{How would an intelligent path-planning be made intelligently?}
\author{Brest State Technical University\\
           Moskowskaja 267, Brest 224017 Republic of Belarus\\
           akira@bsty.by}
\date{}

\begin{document}
\maketitle
\pagestyle{empty}
\thispagestyle{empty}


\begin{abstract}
\noindent
This paper is a contemplation of what it meant by intelligence when we say artificial intelligence. Human intelligence is not always optimizedly efficient but rather spontaneous or unpredictable more or less. Even when we come accross a similar situation as before our behaviour may be different the one than before. To realize such a flexibility in an artificial intelligent agent, we propose a benchmark in which we have infinite nubmer of equally valuable solutions. And then we apply some of machine learning techniques to make an agent behave like more human's way.
\end{abstract}

\section{Introduction}
\begin{quotation}
\begin{small}
\noindent
{\it 
In the "Star Trek" prequel, Spock's father tells him, "You will always be a child of two worlds," urging him not to keep such a tight vise on his emotions. And Spandexy Old Spock, known as Spock Prime, tells his younger self: "Put aside logic. Do what feels right."
-- from New York Times on 10 May 2009 by Maureen Dowd.
}
\end{small}
\end{quotation}

\noindent
When we say a {\it ``path planing problem,''} we usually concerned with finding a {\it shortest path} from a specified point to be started to a specified point of destination. But what if our concern is the {\it longest path} with a limited amount of energy to reach the goal? While shortest path problem has a unique solution, the longest path problem has multiple solutions.\\

\noindent
One option to realize this idea is learning during actions.
We now look at such trial with artificial neural networks.
Floreano et al (2000) showed one of such trials
later Stanley (2003) united Floreano et al.'s 4 rules into one equation with two parameters.
Durr (2008) proposed a more general equation of learnign with
All of these experiment were made using neurons whose statecontinuous
Here we try neural network with spiking neuron.
A counter part of Hebbian learning then is STDP.






\section{Experiments}
\bigskip \noindent
{\bf Challenge (Jeep's survival in a desert)}
{\it In a gridworld, agent moves to the neighboring cell spending one unit of energy. 
The gridworld is sufficiently large so that agent never reaches the border.
Starting from the base located in the center of the gridworld with $N$ units of energy, an agent must travel visiting as many different cells as possible,  and the agent must go back to the base before consuming all the energy.} 

\noindent
Try to imagine a londrover in a planet like the Mars. The mission of the rover is to explore the
areas which have not yet explored so far with a limited energy charged when it starts the base.

\subsection{Random wolk as a base of comparison}


\subsection{Genetic Algorithm with a heuristic}
\begin{figure}[h]
\begin{center}
\includegraphics[width=10cm,height=6cm]{rover-300.eps}
%%\includegraphics[width=6cm,height=6cm]{navi-1-new.eps}
\end{center}
\end{figure}

\subsection{Reinforcement Learning}
In a standard implementation, agent is given a reward only when it reaches to the goal. 
Hecne, as ... put it
{\it "Using random exploration through the search space, rewards may simply never be encountered."} %%KURT DRIESSENS and SASO DZEROSKI (2004) "Integrating Guidance into Relational Reinforcement Learning" Machine Learning, 57, pp.271-304,
This might be a reason why usualy those proposed world to be explored include obstacles
And agent is given a penalty when it touches an obstacle or. 
In a sense,obstacles are not obstacle. Though the agent should avoid obstacles but at the same time those obstacles are good guides to lead the navigation.
Our gridworld however has no border nor obstacle and includes only one goal to be reached.
which make the search {\it a-needle-in-a-haystack-problem}.

Our trick is, (i) agent is given a reward at any cell it passes. 
(ii) reward is associated with the manhattan distance from the starting point
(iii) the further away the agent goes, the more reward agent obtains when its first half of the travel and vice versa when its the last half of the travel. 

\subsubsection{A random walk for comparison}

\begin{figure}[h]
\begin{center}
\includegraphics[width=10cm,height=6cm]{reinforcement-300.eps}
%%\includegraphics[width=6cm,height=6cm]{navi-1-new.eps}
\end{center}
\end{figure}

\subsection{Are we happy with those results?}
In GA a chromosome indicates how agent should move. IN Reinfocement learning
a policy indicates. As a result the behaviour is deterministic.
Agent behaves any trials afterwords as exactly similar as when it learns.
However efficient the behavior might be can we call a intelligent?

\subsection{A newral network approach toward an intelligent behavior}



\subsubsection*{Acknowledgment}
This paper is based on the talk I made in the Seminar .... 


\subsubsection*{Reference}
\begin{description}
\item
Floreano, D., and J.~Urzelai (2000)
``Evolutionary robots with online self-organization and behavioral fitness.''
Neural Networks Vol. 13, pp.~431--434.

\item
Stanley, K.~O., B.~D. Bryant, and R.~Miikkulainen (2003)
``Evolving adaptive neural networks with and without adaptive synapses.''
Proceedings of the IEEE Congress on Evolutionary Computation, pp.~2557--2564.

\item
Florian, R.~V. (2007)
``Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity.''
Neural Computing, Vol.~19, No.~6. pp.~1468-1502.

\end{description}
\end{document}
