Analysis of Multilayer Perceptron with Rectifier Linear Unit Activation Function

Analysis of Multilayer Perceptron with Rectifier Linear Unit Activation Function

Meirambek Mukhametkhan, Olga Krestinskaya and Alex Pappachen James
Electrical and Computer Engineering Department
Nazarbayev University, Astana, Kazakhstan
Abstract

The implementation of analog neural network and online analog learning circuits based on memristive crossbar has been intensively explored in recent years. The implementation of various activation functions is important, especially for deep leaning neural networks. There are several implementations of sigmoid and tangent activation function, while the implementation of the neural networks with linear activation functions is an open problem. Therefore, this paper introduces a multilayer perceptron design with linear activation function. The temperature and noise analysis was performed. The perceptron showed a good performance and strong durability to temperature changes.

Multi layer perceptron, crossbar, memristor

I Introduction

The discovery of the memristive elements lead to the possibility to implement various neural networks on hardware [1, 2, 3, 4, 5]. There are several recent works that illustrate the implementation of analog activation functions for neural networks [6, 4, 1, 7, 2]. Most of the works illustrate implementation of basic perceptron based neural networks with sigmoid and tangent activation functions [6, 4], while the possibility of neural network with other activation functions is not explored.

In this paper, we illustrate the implementation of the basic perceptron neural network with linear activation function. We provide a basic information about the performance of the memristor and show the simulation of the memristor behavior. By applying different amplitude of voltage desired resistance can be obtained. Below threshold voltage memristor acts like a simple resistor with it last set resistance. During simulations, memristor was set to different resistance such as 40, 17 . Based on the performance of the memristor, we implement the programming cells of the 2x2 crossbar array and build a simple perceptron network with linear activation function circuit. Temperature and noise analysis were performed on perceptron circuit. The advantages of memristor-based circuit in terms of area and power dissipation are discussed.

Ii Memristor Model

This section provides some background information on memristor along with its simulation, and it provides some discussion on different operating ranges of simple memristor circuit.

Ii-a What is memristor?

Three basic two terminal components, namely, resistor, capacitor and inductor, are related with four fundamental variables: voltage, current, charge, and flux[8]. The fourth component known as memristor, introduces relation between charge and flux, d = Mdq, where M is defined as memristance[9]. The name memristor(memory resistor) is chosen because resistance of the memristor defined by the history of current passing through it[8].

Fig. 1: Memristor testing circuit

Ii-B Memristor simulation

AC supply voltage is applied to memristor to analyze memristor behavior. Memristor has its threshold voltage below which operating resistance does not change[8]. In practice, different characteristic curves were obtained by altering the amplitude of voltage supply. Initial resistance of memristor was set to 50

Ii-C Results

It is noticed that memristor shows two different behaviors depending on variable voltage. As can be seen from Fig. 2, for some values of voltage, current changes linearly and has non-linear behavior in some regions. This can be explained by the equations (1) and (2) given in report[9]. This means that above threshold voltage memristor resistance changes non-linearly.

(1)
(2)

where is mobility.

Fig. 2: Transient analysis of source voltage, and memristor current

Furhter analysis of memristor show that as the input voltage increases, current curve gets non-linear. In Fig. 3, four different current curves are plotted. One which has sharpest change correspond to 1.5 volt input and means that higher theamplitude of input voltage sharper the change in resistance.

Fig. 3: Transient analysis of memristor current, , behavior for different input voltages.
Fig. 4: I-V characteristic curve of memristor

As can be seen from the Fig. 4, it is clear that threshold voltage is 1V and above this point I-V curve is non-linear. Below 1V, two straight lines are obtained. Line with larger gradient represents ON state of the memristor. Similarly, second line represent OFF state as the resistance is high and current passing through the memristor is low.

Fig. 5: Resistance setting voltages

By applying right duration or amplitude of pulse as mentioned before, desired resistance can be obtained. The formula for resistance estimation can be found in sub file of memristor given in appendix which is used in [10, 11] and discussed in report [11]. Fig. 5 exactly labels the change on resistance of the memristor in terms of voltage amplitude accross the memristor. All pulses have duration of 0.2s. Knowing that and are 62 and 3 respectively following table I was drawn.

Voltage amplitude, V Obtained resistance, k
1 Remains the same
1.4 17
2 3
2.5 3

TABLE I: Resistance programming with input voltage

Iii Multi Layer Perceptron

Perceptron, also termed as single layer perceptron, is a simple artificial (hardware or software) model of biological neuron and synapses. The idea of perceptron is to give single analog or binary output based on several analog or binary inputs [12].

Iii-a Crossbar Array

Crossbar array could be used as a single or multi layer perceptron depending on its dimension. For example, 2x2 and 3x3 arrays correspond to two and three layer perceptrons, respectively. In Fig. 7 2x2 crossbar array is shown. Each column represents a layer and memristors play a role of weighted links(synapses).

Fig. 6: Mutilayer Perceptron

In this kind of array write and read operations can be implemented. So the perceptron can be trained with the feedback system. To write on chosen cell voltage applied to desired column and corresponding ground should be grounded. Side effect of write operation is that extra current flows appear on other cells. For example Figs. 7 and 8 show regulated memristor with blue color and unwanted current flowing through memristors M2 and M3 labeled with red.

Fig. 7: Crossbar array

Furthermore, in Fig. 9, 3x3 crossbar array shown with chosen cell which is programmed and cells with extra current labelled with red. Extra current in red dots shown in figures can cause noise errors and lead to extra power consumption. Possible solutions of this problem are given in [13] with adding MOSFETS or diodes. After the resistance of each memristor is set until last, Information(resistance) is read applying voltage below threshold. In Figs. 10 and 11 the same behavior as for single memristor can be noticed for the resistance of memristor which is being programmed. Fig. 12 shows current flow in each of the memristor in 2x2 crossbar array. If voltage below threshold is applied, should flow through the memristor.

Fig. 8: Crossbar array


(a)

(b) fig 2

(c) fig 3
Fig. 9: Add your own figures before compiling
Fig. 10: Resistance curves of four memristors while write operation
Fig. 11: Current flowing in each memristor of 2x2 array while write operation

Iii-B Rectifier Linear activation function

It was mentioned earlier that perceptron has its current summator and activation function. Circuit for the activation function is shown in Fig. 13. Diode is addded to the output of the opamp to cut the negative current flow in the circuit. Output characteristic of activation function can be observed from Fig. 15. It is clear from figure that output voltage linear dependent on input voltage and has oV output at negative input as it was desired. The circuit in Fig. 13 is also knowns as superdiode.

Fig. 12: Crossbar array with activation function

Vout

Vin

Fig. 13: Linear activation function
Fig. 14: Output voltage and input voltage relation

Iii-C Temperature analysis

Temperature analysis were performed in LTspice for both crossbar array and activation function. During the test between -50 and C any significant change in characteristic curves was noticed. Temperature change had effect only in 2-3 pA range and is hard to differentiate from the graph. This means that, introduced circuit of perceptron has durability to temperature change.

Iii-D Noise analysis and error

Noise test was performed in LTspice and from Fig. 15, it clearly seen that error due to the noise increases as operating frequency increases. Also, it is noted that after 100Hz error distribution gets saturated. In addition, error during write operation of crossbar array increases by the number of write cycles. According to [14], error increases slowly until 50th cycle, then sharply decreases along with the number of cycles is increased further. Also [14] state that, it is difficult to realise 16x16 crossbar array as it has significant error values.

Fig. 15: Noise ration of activation function circuit

Iii-E Area calculation

Memristor is promising device in terms of its size. Nowadays, 1 mosfet transistor occupy the same area as 8 memristors[15]. Investigation done by [13] state that, memristor based memory cells have density 3 times greater than its closest opponent, flash memory. It is 154Gbit.sec for memristor based circuits and 52 for traditional flash memory. In size, again, memristor based memory cells are two times smaller that flash memory where transistors are used.

Iv Conclusion

The study presents deep explanation of memristor simulation which is hardly implemented in memristor based crossbar array. Temperature analysis state that perceptron analyzed in this study has good performance and potential. Noise analysis higher than 100Hz show constant error for any frequency meaning that circuit is durable to noise too. In terms of area memristor has huge advantage over MOSFETs. Small are use can lead to reduced power consumptions.

References

  • [1] O. Krestinskaya, T. Ibrayev, and A. P. James, “Hierarchical temporal memory features with memristor logic circuits for pattern recognition,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2017.
  • [2] A. James, T. Ibrayev, O. Krestinskaya, and I. Dolzhikova, “Introduction to memristive htm circuits,” in Memristor and Memristive Neural Networks, InTech, 2018.
  • [3] O. Krestinskaya and A. P. James, “Feature extraction without learning in an analog spatial pooler memristive-cmos circuit design of hierarchical temporal memory,” Analog Integrated Circuits and Signal Processing, pp. 1–9, 2018.
  • [4] K. Smagulova, O. Krestinskaya, and A. P. James, “A memristor-based long short term memory circuit,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
  • [5] A. James, T. Ibrayev, and O. Krestinskaya, “Design and implication of a rule based weight sparsity module in htm spatial pooler,” in Electronics , Circuits and Systems (ICECS), 2017 24th IEEE International, IEEE, 2017.
  • [6] O. Krestinskaya, K. N. Salama, and A. P. James, “Analog backpropagation learning circuits for memristive crossbar neural networks,” in Circuits and Systems (ISCAS), 2018 IEEE International Symposium on, IEEE, 2018.
  • [7] A. Irmanova and A. P. James, “Neuron inspired data encoding memristive multi-level memory cell,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
  • [8] L. Chua, “Memristor-the missing circuit element,” IEEE Transactions on Circuit Theory, vol. 18, pp. 507–519, September 1971.
  • [9] D. B. Strukov, G. S. Snider, D. R. Stewart, and R. S. Williams, “The missing memristor found,” Nature, vol. 453, pp. 80 EP –, May 2008.
  • [10] C. Yakopcic, T. M. Taha, G. Subramanyam, and R. E. Pino, “Memristor spice model and crossbar simulation based on devices with nanosecond switching time,” in The 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1–7, Aug 2013.
  • [11] D. Biolek, Z. Kolka, V. Biolková, Z. Biolek, M. Potrebić, and D. ToÅ¡ić, “Modeling and simulation of large memristive networks,” International Journal of Circuit Theory and Applications, vol. 46, no. 1, pp. 50–65.
  • [12] F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain.,” Psychological Review, vol. 65, no. 6, pp. 386–408, 1958.
  • [13] M. A. Zidan, H. A. H. Fahmy, M. M. Hussain, and K. N. Salama, “Memristor-based memory: The sneak paths problem and solutions,” Microelectronics Journal, vol. 44, no. 2, pp. 176 – 183, 2013.
  • [14] C. Yakopcic, R. Hasan, and T. M. Taha, “Hybrid crossbar architecture for a memristor based cache,” Microelectronics Journal, vol. 46, no. 11, pp. 1020 – 1032, 2015.
  • [15] H. Abbas, Y. Abbas, S. N. Truong, K.-S. Min, M. R. Park, J. Cho, T.-S. Yoon, and C. J. Kang, “A memristor crossbar array of titanium oxide for non-volatile memory and neuromorphic applications,” Semiconductor Science and Technology, vol. 32, no. 6, p. 065014, 2017.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
207462
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description