Analysis of Multilayer Perceptron with Rectifier Linear Unit Activation Function
Abstract
The implementation of analog neural network and online analog learning circuits based on memristive crossbar has been intensively explored in recent years. The implementation of various activation functions is important, especially for deep leaning neural networks. There are several implementations of sigmoid and tangent activation function, while the implementation of the neural networks with linear activation functions is an open problem. Therefore, this paper introduces a multilayer perceptron design with linear activation function. The temperature and noise analysis was performed. The perceptron showed a good performance and strong durability to temperature changes.
I Introduction
The discovery of the memristive elements lead to the possibility to implement various neural networks on hardware [1, 2, 3, 4, 5]. There are several recent works that illustrate the implementation of analog activation functions for neural networks [6, 4, 1, 7, 2]. Most of the works illustrate implementation of basic perceptron based neural networks with sigmoid and tangent activation functions [6, 4], while the possibility of neural network with other activation functions is not explored.
In this paper, we illustrate the implementation of the basic perceptron neural network with linear activation function. We provide a basic information about the performance of the memristor and show the simulation of the memristor behavior. By applying different amplitude of voltage desired resistance can be obtained. Below threshold voltage memristor acts like a simple resistor with it last set resistance. During simulations, memristor was set to different resistance such as 40, 17 . Based on the performance of the memristor, we implement the programming cells of the 2x2 crossbar array and build a simple perceptron network with linear activation function circuit. Temperature and noise analysis were performed on perceptron circuit. The advantages of memristorbased circuit in terms of area and power dissipation are discussed.
Ii Memristor Model
This section provides some background information on memristor along with its simulation, and it provides some discussion on different operating ranges of simple memristor circuit.
Iia What is memristor?
Three basic two terminal components, namely, resistor, capacitor and inductor, are related with four fundamental variables: voltage, current, charge, and flux[8]. The fourth component known as memristor, introduces relation between charge and flux, d = Mdq, where M is defined as memristance[9]. The name memristor(memory resistor) is chosen because resistance of the memristor defined by the history of current passing through it[8].
IiB Memristor simulation
AC supply voltage is applied to memristor to analyze memristor behavior. Memristor has its threshold voltage below which operating resistance does not change[8]. In practice, different characteristic curves were obtained by altering the amplitude of voltage supply. Initial resistance of memristor was set to 50
IiC Results
It is noticed that memristor shows two different behaviors depending on variable voltage. As can be seen from Fig. 2, for some values of voltage, current changes linearly and has nonlinear behavior in some regions. This can be explained by the equations (1) and (2) given in report[9]. This means that above threshold voltage memristor resistance changes nonlinearly.
(1) 
(2) 
where is mobility.
Furhter analysis of memristor show that as the input voltage increases, current curve gets nonlinear. In Fig. 3, four different current curves are plotted. One which has sharpest change correspond to 1.5 volt input and means that higher theamplitude of input voltage sharper the change in resistance.
As can be seen from the Fig. 4, it is clear that threshold voltage is 1V and above this point IV curve is nonlinear. Below 1V, two straight lines are obtained. Line with larger gradient represents ON state of the memristor. Similarly, second line represent OFF state as the resistance is high and current passing through the memristor is low.
By applying right duration or amplitude of pulse as mentioned before, desired resistance can be obtained. The formula for resistance estimation can be found in sub file of memristor given in appendix which is used in [10, 11] and discussed in report [11]. Fig. 5 exactly labels the change on resistance of the memristor in terms of voltage amplitude accross the memristor. All pulses have duration of 0.2s. Knowing that and are 62 and 3 respectively following table I was drawn.
Voltage amplitude, V  Obtained resistance, k 

1  Remains the same 
1.4  17 
2  3 
2.5  3 

Iii Multi Layer Perceptron
Perceptron, also termed as single layer perceptron, is a simple artificial (hardware or software) model of biological neuron and synapses. The idea of perceptron is to give single analog or binary output based on several analog or binary inputs [12].
Iiia Crossbar Array
Crossbar array could be used as a single or multi layer perceptron depending on its dimension. For example, 2x2 and 3x3 arrays correspond to two and three layer perceptrons, respectively. In Fig. 7 2x2 crossbar array is shown. Each column represents a layer and memristors play a role of weighted links(synapses).
In this kind of array write and read operations can be implemented. So the perceptron can be trained with the feedback system. To write on chosen cell voltage applied to desired column and corresponding ground should be grounded. Side effect of write operation is that extra current flows appear on other cells. For example Figs. 7 and 8 show regulated memristor with blue color and unwanted current flowing through memristors M2 and M3 labeled with red.
Furthermore, in Fig. 9, 3x3 crossbar array shown with chosen cell which is programmed and cells with extra current labelled with red. Extra current in red dots shown in figures can cause noise errors and lead to extra power consumption. Possible solutions of this problem are given in [13] with adding MOSFETS or diodes. After the resistance of each memristor is set until last, Information(resistance) is read applying voltage below threshold. In Figs. 10 and 11 the same behavior as for single memristor can be noticed for the resistance of memristor which is being programmed. Fig. 12 shows current flow in each of the memristor in 2x2 crossbar array. If voltage below threshold is applied, should flow through the memristor.
IiiB Rectifier Linear activation function
It was mentioned earlier that perceptron has its current summator and activation function. Circuit for the activation function is shown in Fig. 13. Diode is addded to the output of the opamp to cut the negative current flow in the circuit. Output characteristic of activation function can be observed from Fig. 15. It is clear from figure that output voltage linear dependent on input voltage and has oV output at negative input as it was desired. The circuit in Fig. 13 is also knowns as superdiode.
IiiC Temperature analysis
Temperature analysis were performed in LTspice for both crossbar array and activation function. During the test between 50 and C any significant change in characteristic curves was noticed. Temperature change had effect only in 23 pA range and is hard to differentiate from the graph. This means that, introduced circuit of perceptron has durability to temperature change.
IiiD Noise analysis and error
Noise test was performed in LTspice and from Fig. 15, it clearly seen that error due to the noise increases as operating frequency increases. Also, it is noted that after 100Hz error distribution gets saturated. In addition, error during write operation of crossbar array increases by the number of write cycles. According to [14], error increases slowly until 50th cycle, then sharply decreases along with the number of cycles is increased further. Also [14] state that, it is difficult to realise 16x16 crossbar array as it has significant error values.
IiiE Area calculation
Memristor is promising device in terms of its size. Nowadays, 1 mosfet transistor occupy the same area as 8 memristors[15]. Investigation done by [13] state that, memristor based memory cells have density 3 times greater than its closest opponent, flash memory. It is 154Gbit.sec for memristor based circuits and 52 for traditional flash memory. In size, again, memristor based memory cells are two times smaller that flash memory where transistors are used.
Iv Conclusion
The study presents deep explanation of memristor simulation which is hardly implemented in memristor based crossbar array. Temperature analysis state that perceptron analyzed in this study has good performance and potential. Noise analysis higher than 100Hz show constant error for any frequency meaning that circuit is durable to noise too. In terms of area memristor has huge advantage over MOSFETs. Small are use can lead to reduced power consumptions.
References
 [1] O. Krestinskaya, T. Ibrayev, and A. P. James, “Hierarchical temporal memory features with memristor logic circuits for pattern recognition,” IEEE Transactions on ComputerAided Design of Integrated Circuits and Systems, 2017.
 [2] A. James, T. Ibrayev, O. Krestinskaya, and I. Dolzhikova, “Introduction to memristive htm circuits,” in Memristor and Memristive Neural Networks, InTech, 2018.
 [3] O. Krestinskaya and A. P. James, “Feature extraction without learning in an analog spatial pooler memristivecmos circuit design of hierarchical temporal memory,” Analog Integrated Circuits and Signal Processing, pp. 1–9, 2018.
 [4] K. Smagulova, O. Krestinskaya, and A. P. James, “A memristorbased long short term memory circuit,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
 [5] A. James, T. Ibrayev, and O. Krestinskaya, “Design and implication of a rule based weight sparsity module in htm spatial pooler,” in Electronics , Circuits and Systems (ICECS), 2017 24th IEEE International, IEEE, 2017.
 [6] O. Krestinskaya, K. N. Salama, and A. P. James, “Analog backpropagation learning circuits for memristive crossbar neural networks,” in Circuits and Systems (ISCAS), 2018 IEEE International Symposium on, IEEE, 2018.
 [7] A. Irmanova and A. P. James, “Neuron inspired data encoding memristive multilevel memory cell,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
 [8] L. Chua, “Memristorthe missing circuit element,” IEEE Transactions on Circuit Theory, vol. 18, pp. 507–519, September 1971.
 [9] D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams, “The missing memristor found,” Nature, vol. 453, pp. 80 EP –, May 2008.
 [10] C. Yakopcic, T. M. Taha, G. Subramanyam, and R. E. Pino, “Memristor spice model and crossbar simulation based on devices with nanosecond switching time,” in The 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1–7, Aug 2013.
 [11] D. Biolek, Z. Kolka, V. BiolkovÃ¡, Z. Biolek, M. PotrebiÄ, and D. ToÅ¡iÄ, “Modeling and simulation of large memristive networks,” International Journal of Circuit Theory and Applications, vol. 46, no. 1, pp. 50–65.
 [12] F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain.,” Psychological Review, vol. 65, no. 6, pp. 386–408, 1958.
 [13] M. A. Zidan, H. A. H. Fahmy, M. M. Hussain, and K. N. Salama, “Memristorbased memory: The sneak paths problem and solutions,” Microelectronics Journal, vol. 44, no. 2, pp. 176 – 183, 2013.
 [14] C. Yakopcic, R. Hasan, and T. M. Taha, “Hybrid crossbar architecture for a memristor based cache,” Microelectronics Journal, vol. 46, no. 11, pp. 1020 – 1032, 2015.
 [15] H. Abbas, Y. Abbas, S. N. Truong, K.S. Min, M. R. Park, J. Cho, T.S. Yoon, and C. J. Kang, “A memristor crossbar array of titanium oxide for nonvolatile memory and neuromorphic applications,” Semiconductor Science and Technology, vol. 32, no. 6, p. 065014, 2017.