Perceptron Linear Function Design with CMOS-Memristive Circuits
In the last decade, the interest to emulation of the functionality and structure of the human brain to solve the problems related to image processing and pattern recognition, especially using to Artificial Neural Network (ANN), has increased. Since the capability of ANN to compute at high-speed has been proven to be very useful for various computational problems. One of the simple ANN models is perceptron. Since the perceptron is the basic form of a neural network, the efficient implementation of analog activation functions is required. As various works introduce the design of sigmoid and tangent activation functions, the other activation functions remain an open research problem. This paper describes the design of the perception circuit with the linear activation function using operational amplifier and memristive crossbar. Additionally, the variation of performance with temperature, noise of the circuit is presented.
Pattern recognition, classification, control, and optimization are the main applications of the artificial neural network . Performance and ability to make decision of ANN’s mostly depend on type of the activation function and network structure. Therefore, to simplify the structure of the network, accelerate the time of convergence and the learning outcome of ANN it is necessary to design activation function that will meet these properties . Various activation functions for memristive crossbar-based neural network designs have been investigated in the last decades [3, 4, 5, 6, 7, 8]. However, the implementation of the memristive neural network with linear activation function is still an open problem.
In this paper, we present the design of the linear activation function for the memristive crossbar perceptron circuit. We explore the possibility to use the linear activation function called Rectified Linear Unit (ReLU) in simple neural network design. Easier optimization, fast convergence, and high computation speed in deep networks are the main advantages of the ReLU .
The simplest model of a neuron is illustrated in the Fig 2, where the in each neuron weighted inputs are summed up in order to generate output signal.
The output of the simple neuron can be simply written in the form given in equation (2).
Where and is the input and weight of the synapse, respectively and the function is the activation function of the neuron .
Some of the neural networks use memristive crossbar technology in processors, where most of these memristive crossbars can be used to compute the dot product by consuming less power and can be used as storage for the weights. The implementation of the memristive circuits illustrated in Fig. 2 is quite simple. The inputs (,, ) are voltage sources connected to respective memristors (, , ), where memristors represent the weight of synaptic connections. Conductance of the memristor is multiplied with input voltage which is proportional to the currents flowing through synapses. Then, by Kirchhoff’s current law all currents are summed up and the output sum is evaluated by activation function .
The main purpose of this paper is to introduce the perceptron with Rectified Linear Unit activation function using memristive crossbar as the storage for synaptic weights.
In this part of the paper memristor device will be described. In the early 1970s it was scientifically proved that there exist fourth basic circuit element that illustrate the relationship between flux and charge called . The name of the device comes from its behavior, which similar to nonlinear resistor which has a memory . The main characteristics of the device can be observed by connecting the memristor to AC source.
Conceptual view of memristor (Fig. 3), demonstrated by HP Lab in 2008, is equivalent to serially connected variable resistors. Where, high resistance state and low resistance state is denoted by and . Expression of the overall memristance of the device is shown in equation (3).
Where front position relative doping is the ratio between doping front position and thin film thickness .
From the Fig. 4 it can be seen that the resistance of the device changes since I-V curve is not linear. However, device needs certain threshold voltage in order to change its state , which is equal to 1V in our model. That means the resistance of the memristor will stay constant until applied voltage will not exceed 1 volt. Also, from the Fig. 4 it can be observed that the I-V graph has three linear parts illustrating three different resistance stages. First one, which was observed in 0.8V and 1V source amplitude is , second one with higher slope is and the last is . According to the sub file in LTSpice the resistance values of illustrated model is , and .
Iii-a Perceptron design
One of the main purposes of this work is to illustrate the circuit of perceptron with linear activation function. ReLU was chosen as an activation function due to its advantages compared to other activation functions. Transfer characteristics similar to ReLU function was observed in precision half wave rectifier circuit also known as superdiode (Fig. 5).
Circuit can be analyzed by considering two cases, when and . For , current , hence ,output voltage will be equal to . For , the current will be negative, however, due to diode it will be blocked because diode will be in reverse biased mode. As a result, output voltage will be zero.
The transfer characteristics of the circuit obtained during simulation is shown in Fig. 6.
From the Fig. 6 it can be seen that the upper bound of the transfer characteristics is saturated. However, according to equation provided in 2 it should not. Further increase in the voltage value leads to increased current value. Since the memristor crossbar is implemented in nano level it is preferred to have lower values of current to prevent overheating and power dissipation.
Iii-B Noise and temperature resistance
It is always preferred to check the implemented circuit for error that can be caused by noise and temperature. Proposed neuron circuit with several memristors was simulated in LTSpice for noise and temperature error. Simulation results showed that in the region of temperature from to the circuit works very well. Also, noise was added to input sources in the frequency range from 10 Hz to 100MHz. Error caused by noise in the output of the circuit was 160 , which can be considered as negligible change. The effect of noise on circuit is shown in Fig.8.
Iii-C Area and power calculation
Before starting manufacturing any device the area of the device or circuit is calculated. Since the memristor crossbar is used in perceptron circuit, area calculation causes some difficulties because it will depend on the input size and other factors. However, memristor took 4 times less area than transistors, but calculations was made on 4x4 and 8x8 tiled memristor systems . Further calculations of larger memristor systems is still needed.
The following paper has been proposed the perceptron linear activation function design using operational amplifier with memristive circuits. The activation function was designed using precision half wave rectifier circuit and memristive crossbar. Designed circuit was tested to noise and temperature effects and it was concluded that the circuit is able to withstand the effects of noise and temperature difference. The mathematical analysis of the given design need to be done and it is assumed that the design can be developed further.
-  S. M. Gowda, B. J. Sheu, J. Choi, C. G. Hwang, and J. S. Cable, “Design and characterization of analog vlsi neural network modules,” IEEE Journal of Solid-State Circuits, vol. 28, no. 3, pp. 301–313, Mar 1993.
-  S. Jeyanthi and M. Subadra, “Implementation of single neuron using various activation functions with fpga,” in 2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies, May 2014, pp. 1126–1131.
-  O. Krestinskaya, T. Ibrayev, and A. P. James, “Hierarchical temporal memory features with memristor logic circuits for pattern recognition,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2017.
-  O. Krestinskaya, K. N. Salama, and A. P. James, “Analog backpropagation learning circuits for memristive crossbar neural networks,” in Circuits and Systems (ISCAS), 2018 IEEE International Symposium on. IEEE, 2018.
-  K. Smagulova, O. Krestinskaya, and A. P. James, “A memristor-based long short term memory circuit,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
-  N. Dastanova, S. Duisenbay, O. Krestinskaya, and A. P. James, “Bit-plane extracted moving-object detection using memristive crossbar-cam arrays for edge computing image devices,” IEEE Access, vol. 6, pp. 18 954–18 966, 2018.
-  A. James, T. Ibrayev, O. Krestinskaya, and I. Dolzhikova, “Introduction to memristive htm circuits,” in Memristor and Memristive Neural Networks. InTech, 2018.
-  A. Irmanova and A. P. James, “Neuron inspired data encoding memristive multi-level memory cell,” Analog Integrated Circuits and Signal Processing, pp. 1–6, 2018.
-  M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, and G. E. Hinton, “On rectified linear units for speech processing,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, May 2013, pp. 3517–3521.
-  K. M. Al-Ruwaihi, “Cmos analogue neurone circuit with programmable activation functions utilising mos transistors with optimised process/device parameters,” IEE Proceedings - Circuits, Devices and Systems, vol. 144, no. 6, pp. 318–322, Dec 1997.
-  D. J. Mountain, M. R. McLean, and C. D. Krieger, “Memristor crossbar tiles in a flexible, general purpose neural processor,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 8, no. 1, pp. 137–145, March 2018.
-  L. Chua, “Memristor-the missing circuit element,” IEEE Transactions on Circuit Theory, vol. 18, no. 5, pp. 507–519, September 1971.
-  M. Hu, H. Li, Y. Chen, Q. Wu, G. S. Rose, and R. W. Linderman, “Memristor crossbar-based neuromorphic computing system: A case study,” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 10, pp. 1864–1878, Oct 2014.
-  Y. V. Pershin and M. D. Ventra, “Practical approach to programmable analog circuits with memristors,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 57, no. 8, pp. 1857–1864, Aug 2010.
-  C. Yakopcic, T. M. Taha, and R. Hasan, “Hybrid crossbar architecture for a memristor based memory,” in NAECON 2014 - IEEE National Aerospace and Electronics Conference, June 2014, pp. 237–242.