\chapter{Combining a Volatile and Nonvolatile Memristor in Artificial
Synapse to Improve Learning in Spiking Neural Networks}
\label{synapse-box}
\begin{abstract}

With the end of Moore's law in sight, we need new computing architectures to
satisfy the increasing demands of big data processing. Neuromorphic
architectures are good candidates to low energy computing for recognition and
classification tasks. We propose an event-based spiking neural network
architecture based on artificial synapses. We introduce a novel synapse box that
is able to forget and remember by inspiration from biological synapses. Two
different volatile and nonvolatile memristor devices are combined in the synapse
box.
% We adapt the unsupervised STDP training algorithm to our synapse box.
To evaluate the effectiveness of our proposal, we use system-level simulation in
our Neural Network Scalable Spiking Simulator (N2S3) using the MNIST handwritten
digit recognition dataset.  The first results show better performance of our
novel synapse than the traditional nonvolatile artificial synapses.
\end{abstract}


\section{Introduction}
\label{intro}

Neuromorphic computing has the potential to bring very low power computation to
future computer architectures and embedded systems~\cite{merolla_million_2014}.
Indeed parallel neuromorphic computing, by doing computation and storage in the
same devices can overcome the Von-Neumann bottelneck. Neuromorphic computing is
introduced as an appropriate platform for Big Data analysis and Cloud Computing.
Furthermore, many huge projects are running based-on neuromorphic system such as
the EU Human Brain Project~\cite{markram_human_2012}, the DARPA/IBM SYNAPSE
project~\cite{ananthanarayanan_cat_2009} and deep learning research by Google and Facebook among
others.

Recently, emerging devices in nano-scale have demonstrated novel properties for
making new memories and unconventional processing units.  One of those is the
memristor that was hypothetically presented by Leon Chua in
1971~\cite{chua_memristor-missing_1971} and after a few decades, HP was the first to announce the
successful memristor fabrication~\cite{strukov_missing_2008}. The unique properties  in
memristor nano-devices such as, extreme scalability, flexibility because of analog behavior, and ability to
remember the last state make the memristor a very promising  candidate to apply
it as a synapse in Spiking Neural Network (SNN)~\cite{jo_nanoscale_2010}.

In the recent years, there have been several research works using non-volatile
resistive nanodevice as a synapse to build a SNN
hardware~\cite{merolla_million_2014,jo_nanoscale_2010,indiveri_integration_2013}. Forgetting  in
the biological brain is an important key of adaptive computation, as without
forgetting the biological memory soon becomes overwhelmed by the details of
every piece of information ever experienced. Consequently, some studies have
been done using volatile memory as a synapse in brain-like
computing~\cite{alibart_organic_2010, alibart_memristive_2012,desbief_low_2015}. In this work,  we combine both
volatile and non-volatile types of artificial synapses. It leads to make a
synapse which can forget if the information is not important as well as remember
if it is significant data.

Thanks to close collaboration with the nano-electronics research center in the
University of Lille (IEMN), we have the opportunity of studying the suitability
of different kinds of memristors (TiO$_{2}$, NOMFET, magnetoresistive,
magnetoelectric)  to build a spiking neural network hardware platform.  Due to
the demonstrated potential of NOMFET (Nanoparticle Organic Memory Field-Effect
Transistor)~\cite{alibart_organic_2010, alibart_memristive_2012} to play the role of a synapse, we use it as a
volatile synapse in neuromorphic accelerator. The non-volatile device could be
any solid-state memristor. We have chose here the resistive memory presented
in~\cite{querlioz_immunity_2013} as non-volatile memory.

% It is widely believed that plasticity is the key of learning in biological
% brain~\cite{STDP1}. With the latest proposals to use memristive nano-devices as
% synapses,  we implement efficient unsupervised learning rules like Spike Timing
% Dependent Plasticity (STDP)~\cite{STDP2}.

We evaluate the synapse box proposal by comparing it with a single non-volatile memory synapse
by simulation on the MNIST handwritten digit recognition benchmark. We
use Leaky Integrate and Fire (LIF) neurons in Restricted Boltzmann Machine
(RBM) network topology.
% The contribution is to use this platform and make a
% neuromorphic accelerator suitable for parallel approximated and cognitive
% computing.
% We use analytical models for solving the neuron equation model. Indeed, solving
% the differential equation with numerical methods is too costly in large scale
% networks.
To run the simulations, we introduce the Neural Network Scalable Spiking
Simulator (N2S3), a simulation framework for architecture exploration of
neuromorphic circuits.

%  To
% be able to perform energy efficient computing, we use an event-driven method for
% computing, processing only occurs at reception of a spike. Our key architectural
% abstraction is a network of spiking neurons that process spikes in an efficient,
% scalable, and flexible way.  We use real parameters of fabricated devices to
% ensure that our simulations are meaningful with respect to the technology. The
% platform is not only flexible to analyze different models of plausible neurons,
% synapses and network topologies, but also it is scalable to distribute the
% simulation on several computers.

% To evaluate and verify the effectiveness of learning algorithms and the
% functionality of the neuromorphic platform, we have used the MNIST handwritten
% digit dataset.

In the next section we describe the architecture including the neuron and
synapse models, as well as the network topology and the unsupervised training
algorithm.  Then, in Section~\ref{validation} we present the experimental
evaluation of the two different synapses using the MNIST handwritten digit
dataset and the new spiking neural network simulator we propose, N2S3.

\section{Circuit Design of Neuron and Synapse in RBM Network}
\label{design}
In the biological brain, neurons are computing units. Here we define our
computing unit by inspiration from a biological neuron model. The synapse
operates as a plastic controller between two neurons. The manufacturability
is out of scope of this research, however in the synapse box, we have applied the 
real parameters of volatile
synapse beside the model of nonvolatile Resistive RAM. As there are varieties of 
nonvolatile memristor fabrications,
finding appropriate one that is compatible with NOMFET seems not too complicated.
It worth mentioning that most of the
nanodevices have been reported as Resistive RAMs (nonvolatile) as well as NOMFET
(volatile) are compatible with
CMOS circuits \cite{alibart_memristive_2012}.
\subsection{Leaky Integrate-and-Fire neurons}

The Leaky-Integrate-and-Fire (LIF) neuron model is a well-studied model of neuron.
We discussed it in this dissertation in \ref{ch3:lif}. We have applied this 
model of LIF neurons in our work. If there is enough spikes in the short period of 
time which depend on the resistor and capacitor value, the neuron reaches to the 
threshold. Vice versa, if there is no or small number of spikes connected to input 
of neuron the action potential decrease and neuron is in leaky phase.
\subsection{New artificial synapse using memristors}
Here, we define our computing unit by inspiring from biological neuron model. 
The synapse operates as a plastic controller 
between two neurons. This plasticity is believed
as origin of learning and memory in brain. 
Figure \ref{IMP1} shows a simple circuit of two neurons 
connected with an adaptive synapse (memristor).

\begin{figure}[ht]
\centering
  \includegraphics{./figures/ch3/Neuron-synapse_LIF-1}
 \caption{\textbf{a)}  Schematic of two simple biological neurons connected with synapse,
  \textbf{b)} Leaky Integrated \& Fire model of neuron connected with artificial synapse (memristor) }\label{IMP1}
\end{figure}

Before the discovery of a memristor nanodevice, by using state-of-the-art
technology, 12 transistors were combined to mimic the behavior of memristor to
perform the STDP learning method~\cite{Indiveri-Bistable-syn}. Therefore, using
a two-terminal and scalable device such as the memristor could save remarkable
amount of power and cost specially in modeling large scale Spiking Neural
Networks. To model biological synapses, not only do we need a device to be able
to store the last activity, but it must also have enough flexibility to
achieve Spike Timing-Dependent Plasticity (STDP) for learning. Using memristor
as a nonvolatile synaptic memory has been proposed in several
works~\cite{alibart_organic_2010,jo_nanoscale_2010, kim_memristor_2012, indiveri_integration_2013-1}. 
By using nonvolatile
memory, we can guarantee to store the last synaptic weight which is necessary
for network training but the synapse can not forget. To be able to have a
synapse which is able to forget, scientists used a volatile memory
cell~\cite{alibart_memristive_2012,desbief_low_2015}.

\subsection{New plasticity learning method}
Forgetting is a memory mechanism that helps brain having better functionality.
In fact, it is believed that forgetting helps the brain to remember. However,
remembering details of many daily activities and information such as shopping
list, novel book details or newspapers not only are unnecessary
to remember but also might interfere with brain functionality for innovative
thinking and data analysis. Basically human brain skips details of insignificant
information and remembers the most important, unique and surprising events and
information. In neuroscience, memorization is believed to achieve as a result of
two types of synaptic plasticity: Short-Term Potentiation (STP) and Long-Term
Potentiation~\cite{martin_synaptic_2000}. STP is achieved through the temporal changing of a
synaptic connection and the decrease to its initial state soon after. In LTP,
stimulation iteration causes permanent synaptic weight achievement as it is
depicted in Figure~\ref{fig:LIF-stp}. Shorter
iteration interval leads to more efficient LTP. By inspiring of this biological
theory, we propose a new artificial synapse with the ability to forget
insignificant data while storing significant information. The novel synaptic box
includes one organic transistor and one resistive RAM.

\begin{figure}[ht]
\centering
\includegraphics[scale=1.3]{./figures/ch3/STP-LTP}
  \caption{Memorization inspired from biology,
 the data is stored in Long-Term Memory (LTM) if the spikes are repeated in a certain time-window, 
 otherwise Short-Term Memory (STM) will store temporary data.}
\label{fig:LIF-stp}
\end{figure}
\subsection{Combining a volatile and nonvolatile memristor to make a new artificial synapse}
Resistive RAM is modeled in our previous work~\cite{shahsavari_unconventional_2015} 
and is used here 
as a nonvolatile memristor in the synapse box. As it is shown in
Figure~\ref{SynPhoto}.b by changing the doped-undoped regions of device, the
conductance will be changed. Bigger doped region leads to more conductivity.
Therefore by controlling this boundary between two regions, the conductivity is
controlled. The behavior of memristor can be modeled as
follows~\cite{strukov_missing_2008}:
\begin{align}
v(t)&=R_{m}i(t) \\
R_{m}&={R_{ON}}\frac{{w(t)}}{D} + {R_{OFF}}\left(1 - \frac{{w(t)}}{D}\right)\label{mem-non}
\end{align}
where $R_{m}$ is the variable resistance of memristor, $w(t)$ is the width of
the doped region, $D$ is the overall thickness of device, $R_{ON}$ and $R_{OFF}$
are device resistances while the active region is completely doped ($w=D$) and
mostly undopped ($w \rightarrow 0$) respectively (Figure~\ref{SynPhoto}.b). To
model the changing of the conductance, we use the model extracted from
Equation~\ref{mem-non} and introduced in~\cite{querlioz_learning_2011} by
considering $g_{max}=\frac{1}{R_{ON}}$ and $g_{min}=\frac{1}{R_{OFF}}$ as the
maximum and minimum device conductance respectively.

Organic synaptic memory is a novel memristive device with capability of
mimicking synaptic properties especially forgetting ability that we discussed it in Section \ref{ch1}. 
Here, we use a Nano-particle
Organic Memory Field Effect Transistor (NOMFET) as an organic memristive device
made of conjugated molecules and metal nanoparticles (NPs) which is fabricated
by the Institute of Electronics, Microelectronics and Nanotechnology (IEMN) at Lille
university~\cite{alibart_organic_2010}. We use NOMFET as our volatile device in the
synapse box. In the most recent fabrication process~\cite{desbief_low_2015}, NOMFET works at 1~V
with a typical response time in the range 100--200~ms. NOMFET is designed
particularly for neuro-inspired computing architectures~\cite{alibart_memristive_2012}.
NOMFET uses charge trapping/detrapping in an array of gold nanoparticles (NPs)
with the SiO$_{2}$/pentacene interface designed to mimic dynamic plasticity of a
biological synapse as depicted in Figure~\ref{SynPhoto}~\cite{alibart_memristive_2012}.
The NOMFET is used as a two-terminal device by connecting drain (D) and gate (G)
together and using this terminal as an input. The source (S) is used as output
of the device. Equation 5 shows the behavior of NOMFET as a memristor:
\begin{equation}
 i_{ds}(t)=g(q_{np}(t),v_{ds}(t),t)v_{ds}
\end{equation}
where $g$ is the conductance of the device, $v_{ds}(t)$ is the applied voltage and
$q_{np}$ is the charges trapped in the NP. For more details of physical
structure and behavior of NOMFET refer to~\cite{alibart_memristive_2012,desbief_low_2015}.

Figure~\ref{SynPhoto}.c is the synapse box schematic that we apply in our
simulation platform to take the advantages of both nonvolatile and volatile
artificial synapses. The equivalent circuit of transistor is depicted in
Figure~\ref{SynPhoto}.d. Actually, weight modification follows the STP rule until
reaching the LTP threshold in NOMFET. The modification of nonvolatile device is
based on STDP learning. Indeed the NOMFET reacts similar to a high-pass filter
(HPF). The stimuli spikes with low frequency are not qualified to pass in
forgetting Phase. In LTP, stimuli spikes which have more frequency pass to
interfere in learning phase (Figure~\ref{fig:LIF-stp}).
\begin{figure}\centering
  \includegraphics[scale=1.2]{./figures/ch3/Volatile-nonvolatile}
 \caption{Artificial synapse:
 \textbf{a)} Schematic view of the NOMFET as a volatile memory,
 \textbf{b)} TiO$_{2}$ based nonvolatile memory,
 \textbf{c)} Synapse box schematic,
 \textbf{d)} Equivalent circuit with simple elements}\label{SynPhoto}
\end{figure}

\subsection{Network topology and learning}
By using unsupervised learning inspired by biological neural networks, we propose
a fully connected network architecture similar to Restricted Boltzmann
Machine~\cite{salakhutdinov_restricted_2007}. To figure out the correlation between the data, 
STDP helps
to adjust the weight if the sensory input spikes are frequent enough to pass the
STP and remain in LTP phase. In STDP, if there is output spike in pre-synaptic
neuron and shortly after in post-synaptic neuron, the conductance of the synapse
between two neurons increases. On the other hand, if the post-synaptic neuron
spikes shortly before the pre-synaptic neuron, the conductance of synapse
between two neurons decreases. For more comprehensive explanation for STDP, we refer you to 
Chapter \ref{ch_neuromorphic}. To see how plasticity in
memristor helps targeting STDP achievement we refer you to~\cite{serrano-gotarredona_stdp_2013}.

The simulator architecture in our work is event-driven, there is no clock to
synchronize the inputs and outputs. Furthermore, by inspiring of biological
behavior of brain computing, we apply lateral inhibition to reduce the activity
of the neighbors of winner neurons. This method is known as winner-take-all (WTA)
strategy~\cite{nessler_stdp_2009}. The neuron which reaches the threshold first sends an
inhibitory signal to all other neurons in the same layer to reset their states
during inhibition time.

\section{Experimental Validation}
\label{validation}
In order to check the effectiveness of the synapse, we propose a spiking neural
network simulator. Our requirements for the simulator are: speed (thus
event-driven simulation and concurrency), scalability (thus high-level
abstraction and distributability), and adaptability (possibility to model
different synapses, soma, and network topology). After the presentation of the
simulator, we describe our experimental setting and discuss the simulation
results that show an improvement in recognition rate for the synapse box with
respect to the simple nonvolatile synapse.

\subsection{MNIST recognition improvement}
\label{results}

We have used the MNIST training dataset of handwritten digits \cite{MNIST} to
train and test the performance of neural networks based on the synapse box. The
training set consists of 60000 digits between 0 and 9 and each handwritten
number is a 28 $\times$ 28 pixel image. In this simulation, we present the full
dataset (60000 images) and full images. Each pixel is connected to one input
buffer neuron.
% To transfer the image of a handwritten digit to spikes train, we have tried several spike distributions such as Poisson, Uniform, and Jitter.
% We have choosen the Poisson distribution, however we have not observed remarkable difference regarding to the output results of network efficiency.
Pixel intensity is between 0 to 255 and is transfered to 0 to 22 Hertz spiking
frequency using a Poisson distribution during a 350 ms presentation window.
Based on previous similar work \cite{diehl_unsupervised_2015}, we have chosen a
delay of 150 ms between two images. Therefore, there is sufficient time for
membrane potentials of all neurons to reset back to initial values. The network
connection weights are between 0 and 1 initialized using a Gaussian
distribution.

The hardware platform is a 4 core i7-3687U CPU (2.10GHz $\times$ 4). We have simulated different network topologies consisting of 
2 fully interconnected layers, with a fixed input neuron number ($28\times28=784$) and different output neuron number. 
The neuron model is LIF and we evaluate two types of synapses: non-volatile (NV) and proposed synapse box (volatile/nonvolatile or VNV).
% The synapse box has the responsibility of remembering the last
% synaptic weight as well as filter the data or noises that are required to be forgotten or pass frequent spikes to be remembered.

To measure and evaluate the network classification accuracy after a fully
unsupervised learning period consisting of the presentation of the full MNIST
data set, we label the output neurons using 10000 samples of MNIST: After
training, we stop all synaptic modification processes such as STDP, STP and LTP.
We assign a number class to the output neuron which has most frequent firing
rate during the presentation of the 10000 labelling samples.
% Consequently, with finding highest average firing, we are able to evaluate
% network recognition rate with comparing to input digits. In other words, we keep
% the weights during the training and use them in testing.
Using these labels, we then evaluate the recognition rate of the network on
10000 different test samples by comparing the known class of the input with the
class of the most firing output neuron.
\begin{figure}\centering
  \includegraphics[width=7cm]{./figures/ch3/100n}
 \caption{Synaptic weights (conductance of non volatile memristor) learned in
 simulation using the synapse box with 100 output neurons. The weights in the
 corners are random because they were always filtered out by the volatile
 memristor and thus are never modified or even read.}\label{fig4}
\end{figure}
As it was observed in similar works \cite{diehl_unsupervised_2015} and
\cite{querlioz_immunity_2013}, the recognition rate depends on the number of
neurons and synapses, and the number of repetitions of the presentation of the
dataset to the network.   In the experiments, we present 60000 digit the dataset
the smallest number of time  that is necessary for learning. That is 1 time for
less or equal to 50 output neurons,  2 times for less or equal to 100 output
neurons and 3 times for more output neurons. With these numbers of
presentations, we obtain recognition rates that are comparable to the state of
the art. The  running time of the simulations are also comparable to those of
similar experiments though it is difficult to make accurate comparisons.

An example of the conductance weights learned in N2S3 on the MNIST dataset is
shown in Figure \ref{fig4} for 100 output neurons. As it is obvious in the
figure, the border of each digit did not pass the NOMFET high-pass filter
because of low frequency changes. This is the impact of forgetting properties in
synapse box to skip unimportant data or noise. To demonstrate the functionality
of the synapse box, we compare the recognition rate of networks of the same
topology but using different synapse models: a simple nonvolatile synapse (NV)
model and the volatile-nonvolatile (VNV) synapse box model. We have run 10
simulations for each number of output neuron and each synapse. The results are
summerized in Figure \ref{fig5} showing the distribution of the recognition
rates for each configuration. We can conclude that using the synapse box
improves the recognition rate in average by a small but consistent margin.

Although it is not shown in Figure \ref{fig5}, we have also made the comparison
using 300 output neuron and the best recognition rate we have obtained is
89.4~\%.

\begin {table}
\begin{center}

 \caption{Comparing network architecture efficiency for two synapses: nonvolatile (NV) v.s volatile-nonvolatile (VNV) synapse}

\begin{tabular}{|c|c|c|c|c|}\hline
%\caption{Comparing simple nonvolatile synapse (NS) with volatile-nonvolatile synapse (VNVS)}
\#Output &\#synapses & Training &  NV & VNV  \\
neurons&&repetition&& \\
\hline\hline
 \multirow{ 2}{*}{20}&\multirow{ 2}{*}{15680}& 2 & 67,17\% & 68,73\%\\\cline{3-5}
 &  & 3 & 68,35\%&70,68\%\\\hline
 \multirow{ 2}{*}{30}&\multirow{ 2}{*}{23520}& 2 & 75,05\% & 77,29\%\\\cline{3-5}
&  & 3 & 76,50\%&78,91\%\\\hline
 \multirow{ 2}{*}{50}& \multirow{ 2}{*}{39200}& 2 & 77,79\% & 79,60\%\\\cline{3-5}
&  & 3 & 79,88\%&82,12\%\\\hline
\multirow{ 2}{*}{100}&\multirow{ 2}{*}{78400}& 2 & 82,32\% & 84,83\%\\\cline{3-5}
& & 3 & 85,11\%&86.76\%\\\hline
\multirow{ 2}{*}{300} &\multirow{ 2}{*}{235200} & 2 & 86,23\% & 87,59\%\\\cline{3-5}
 &  & 3 &88,23\%&89,46\%\\\hline
\end{tabular}
\label{Tab-1}
\end{center}
\end {table}

\begin{figure}
\centering
\includegraphics [scale=0.6]{./figures/ch3/nanoarch16plot}
\caption{Recognition rate as a function of number of output neurons. In the
 box plot for each number of neuron, we compare the recognition rate of the two
 synapse models. The whiskers of the box plot represent the minimum and maximum
 recognition rates of the 10 simulations.}\label{fig5}
\end{figure}
 
\section{Conclusion}
\label{summary}

In this study, we have introduced a novel synapse box with the possibility to
forget and remember inspired from biological synapse properties. This synapse
box is  composed of a volatile memristor (NOMFET) followed by a nonvolatile
resistive RAM. The volatile memristor acts like a high-pass filter to enhance
short term plasticity  and the nonvolatile resistive RAM enables long term
potentiation. Both work together in the spike timing dependant plasticity
unsupervized learning process.

In addition, in this work we have also announced and used our event-based simulator,
N2S3 (Neural Network Scalable Spiking Simulator). It is specifically designed to
simulate hardware spiking neural networks. N2S3 is quite flexible to explore
different network architectures, synapse and neuron models to help design
hardware architectures and VLSI circuits. To evaluate and verify the new synapse
box as well as the functionality of the simulator, we have used the MNIST
handwritten digit dataset. The first results demonstrate an improvement in
recognition rate by using the synapse box over a single nonvolatile memristor
synapse. We will continue to explore the various parameters and device
combinations to  help design the most efficient hardware neural networks as
possible.

For future works, we also propose to study different neural network topologies
such as deep belief, recurrent and convolutional neural networks to evaluate the
synapse box benefits and costs (area, energy, manufacturability, variability) in
other contexts.