AC Operation Hardware Neural Circuit and the Design of Deep Learning Model

: In the machine learning field, many application models such as pattern recognition or event prediction have been proposed. Neural Network is typically basic methods of machine learning. Previous analog neural network models were composed of the additional circuit and solid resistance. Additional circuit was realized by operational amplifier. Connecting weights means the solid resistance of circuits. As the reason of the resistance is fixed, changing resistance value and connecting weight is quite difficult. However, in the case of using variable resistance, we have to adjust the resistance value by our hands. In this study, we used analog electronic circuits using alternative current to realize the neural network learning model. These circuits are composed by a rectifier circuit, Voltage-Frequency converter, amplifier, subtract circuit, additional circuit and inverter. The connecting weights describe the frequency, converted to direct current from alternating current by rectifier circuit. The connection weights are able to changed easily compare with another hardware neural model. This model’s architecture is on the analog elements. The learning time and working time are very short because this system is not depending on clock frequency. Moreover, we suggest the realization of the deep learning model about proposed analog hardware neural circuit.


Introduction
In recent years, multi-layered network models, especially deep learning model have been researched very actively.The performance is innovatively improved in the field of pattern / speech recognition.The recognition mechanism is elucidated more and more; self-learning IC chips have also been developed.However, these models are working on a general Von Neumann type computer with the software system.There are only a few studies to construct analog parallel hardware system using biomedical mechanism.In this research, we propose the neural network, machine learning model with pure analog network electronic circuit.This model will develop a new signal device with the analog neural electronic circuit.In the field of neural networks, many practical use models such as pattern recognition or event prediction have been proposed.And there are many hardware implementation models such as vision sensor or parallel network calculator have been developed.

Analog Hardware Neural Network
The main merit of analog hardware neural network is it can operate a continuous time system, not depending on clock frequency.On the other hand, digital system operates depending on clock behaviour on the basement of Von Neumann type computer.As a result, advanced analog hardware model were proposed [1] [2].In the pure analog circuit, one of the task is realization of analog memory, keeping the analog value for a while [3].The DRAM method memorizes in the capacitor as temporary memory, because it can be achieved in the general-purpose CMOS process [4].However, when the data value is kept for a long term, it needs the mechanism to maintain the memory data.For example, refresh process is needed.Capacitor reduce the electric charge with time passage.It is easy to recover the electric charge of capacitor using refresh process in the digital memory.However, in the case of using analog memory, analog refresh process is quite difficult compared digital refresh process of DRAM.Other memorization methods are the floatage gate type device and magnetic substance memories [5][6].

Pulsed Neural Network
Pulsed neural network received time series pluses as learning data and changes the connection weights depends on the number of pluses.This network can keep the connecting weights of network after the learning process and outputs the signal depends on the input value [7].This network can change the connecting weights by pulsed learning signal.However, it needs the long time for learning because many pulses needed before completely learning.For example, about no less than 1mS is needed in the case the pulse interval is 10uS and 100 pulses are received to the network on learning process.

The History of Analog Neural Network
Many research results have been reported in the field of neural network and soft computing.These research fields were having been widely developed not only pattern recognition but also control or forecasting system.The network structure and learning method of a neural network is similar to the biomedical mechanism.The structure of the basic neural network usually consists of three layers.Each layer is composed of the input, connecting weight, summation, thresholds and output unit.In our previous study, we designed and proposed motion detection system or image processing model using multi-layered neural network and artificial retina model.On the other hand, we construct pattern recognition machine using variable resistance, operational amplifier.We used CdS cells in the input sensor.However, we have to adjust the resistance value by our hands.Moreover, capacitor reduce the electric charge with time passage.It needs the analog refresh process.Analog refresh process is quite difficult compared digital refresh process of DRAM.
In the present study, we proposed a neural network using analog multiple circuits and operational amplifier.The learning time and working time are very short because this system is not depending on clock frequency in the digital clock processing.At first we designed a neural network circuit by SPICE simulation.Next we measured the behaviour the output value of the basic neural network by capture CAD and SPICE.Capture is the one of the computer-aided design (CAD) system of SPICE simulation.We compared both output results and confirmed some extent of EX-OR behaviour [10] [11].EX-OR behaviour is typically working confirmation method of three layered neural network model.This EX-OR behaviour is liner separation impossible, it is suitable data for checking neural network ability.

Neural Network using Multiple Circuit
In our previous study, we used multiple circuits to realize the analog neural network.In the SPICE simulation, the circuit is drawn by CAD, called Capture.After setting the input voltage or frequency, SPICE has some analysis function, AC, DC or transient.At first, we made the differential amplifier circuits and Gilbert multiple circuits toward to making the analog neural network.We show the difference circuits in Fig. 1.Many circuits take an input signal represented as a difference between two voltages.There circuits all use some variant of the differential pair.Figure 1 shows the schematic diagram of the simple transconductance amplifier with differential pair.The current mirror formed by Q3 and Q4 is used to form the output current, which is equal to I1-I2.The difference circuits enhanced to two-quadrant multiplier.Its output current can be either positive or negative, but the bias current Ib can be only a positive current.Vb, which controls the current, can be only positive voltage.So the circuit multiplies the positive part of the current Ib by the tanh of (V1-V2).If we plot V1-V2 horizontally, and I vertically, then this circuit can work in only the first and second quadrants.We show the Gilbert multiple circuit in Fig. 2. To multiply a signal of either sign by another signal either sign, we need a four-quadrant multiplier.We can achieve all four quadrants of multiplication by using each of the output currents from the differential pair (I1 or I2) as the source for another differential pair.Figure 2 shows the schematic of the Gilbert multiplier.In the range where the tanh x is approximately equal to x, this circuits multiplies V1-V2 by V3-V4.And we confirmed the    In the previous hardware model of neural network, when we use solid resistance elements, it needs to change the resistance elements with the complex wires in each step of learning process.In the case of using variable resistance, we have to adjust the resistance value by our hands.Figure 3 is the one neuron circuit, using multiple circuit and additional circuit by opamp.Multiple circuit calculate the product of the two input value, input signal and connecting weights.There are three multiple circuits.Two multiple circuits mean two input signal and connecting weights.Other one multiple circuit means threshold part of basic neuron.In the threshold part, input signal is -1.In the multiple circuit, it products input signal -1 and connecting weights.So the output of multiple circuit is threshold of this neuron.

Perceptron Feedback Network by Analog Circuits
In Figure 4, we show the architecture of perceptron neural network.This is basic learning network using teaching signal.'y' means the output signal of neural network.'t' means teaching signal.Error value 't-y' is calculated by subtract circuit.After calculated error value, the product error value and input signal are calculated and making feedback signal.The input of subtract circuit on the feedback line are feedback signal and original connecting weight.This subtract circuit calculates the new connecting weight.After product the new connecting weight, the next time learning process is started.Figure 5 shows the perceptron circuits, two-input and one-output.There are multiple circuits and additional circuits in the feed forward line.Error value between original output and teaching signal is calculated by subtract circuit.There are multiple circuits and additional circuit in the feedback lines.In the experimental result of this perceptron, the learning time is about 900μS shown in Figure 6 [12].Figure 7 shows the Architecture of Three-Layers Neural Circuits.In Figure 8, we show the Learning Neural Circuit on Capture CAD by SPICE.

Neural Circuit on Alternative Current Behaviour
We proposed an analog neural network using multiple circuit in the previous research.However, in the case of constructing a network, one of the disadvantage is that the input and output range is limited.Furthermore, the circuit operation becomes unstable because of the characteristics of the multiple circuit using semiconductor.It is called 'Circuit Limitations'.One of the cause is transistor mismatch.Not all transistors created equal.Another cause is the output-voltage limitation.We tried to use the alternative current as transmission signal in the analog neural network in Figure 9.The original input signal is direct current.We used the voltage frequency converter unit when generate the connecting weight.The input signal and connecting weight generate the Alternative current by the Amplifier circuit.Two Alternative currents are added by an additional circuit.The output of the additional circuit is a modulated wave.This modulated wave is the first output signal of this neural network.When we construct learning AC neural circuit, we have to convert the feed-back modulated current signal to a connecting weight with frequency.Correction error signal is calculated by the products difference signal and input signal.Difference signal is the difference between output value and teaching signal.Figure 12 shows the convergence result of learning experiment.It means the learning process is succeeding with very short time.Figure 13 shows the Basic AC operation learning neural network model.This circuit is composed by a Rectifier circuit, Voltage-Frequency converter, Amplifier, subtract circuit, Additional Circuit and Inverter.The input signal is direct current.The initial value of the connecting weight is also direct current.This direct current is converted to frequency by a Voltage-Frequency converter circuit.The input signal and connecting weight generate the Alternative current by the Amplifier circuit.Two Alternative currents are added by an additional circuit.The output of the additional circuit is a modulated wave.This modulated wave is phase inverted by an Inverse circuit.The phase-inverted wave is amplified.The amplification is the value of the teaching signal.This amplified signal and modulated wave are added by an adder circuit.The output of this adder circuit is the error value, which is the difference between the output and teacher signal.Thus, we do not have to use the subtract circuit to calculate the error value.The output of the Adder circuit is converted to direct current from alternating current by the rectifier circuit.This direct current is a correction signal of connecting weights.New connecting weight is calculated by a subtract circuit.This circuit calculates the original connecting weight and the correction signal of the connecting weight.The output of the subtract circuit is converted to a frequency signal by a voltage-frequency convert circuit.It means that in the AC feedback Circuit for BP learning process, after getting DC current by rectifier circuit, we have to Convert from DC Voltage to AC current with Frequency.Finally, Alternating Current occurs by the amplifier circuit.The amplification is the value of the input signal.Figure 14 shows the simulation results of AC feed-back neural model, two-input signal, connecting weights and after rectified wave.

Deep Learning Model
Recently, a deep learning model has been proposed and developed in many applications such as image recognition and artificial intelligence.Deep learning is a kind of algorithms in the machine learning model.This model is developed in the recent research.The recognition ability is improved more and more.Not only pattern recognition, but also image or speech recognition field, deep learning model is used in the many field in the practical use.And this system is expected in the field of robotics, conversation system and the artificial intelligence.

The Stacked Auto Encoder
The stacked auto-encoder is one of the process in the deep learning.This is the pre-learning method of the large number layer network.In the basic neural network, in the almost case, there are three layers.On the other hand, there are nine layers in the deep learning model generally.After the learning process is completed by machine learning method, remove the decoding part, output layer and the connection of intermediate layer and output layer.Keeping of coded portion means from input layer to the intermediate layer including the connection of input layer and intermediate layer.
Intermediate layer contains the compressed data of input data.Moreover, we obtain more compressed internal representation, as the compressed representation input signal to apply the autoencoder learning.After removing the decoding part of stacked auto-encoder, next network is connected.This network is also learned by another three-layered network and remove the decoding part.
A Stacked auto-encoder has been applied to the Restricted Boltzmann Machine (RBM) as well as the Deep learning network (DNN).Moreover, stacked auto-encoder is used the many types of learning algorithm.Recently, the learning experiment featuring a large amount of extraction from an image has become well-known.Stacked auto-encoder can self-learning of abstract expression data.This network has nine layers with three superimposed sub-networks, such as a convolution network [13].In the previous research, we described the simple neural network learning model by analog electronic circuits.We tried to expand the network to realize the deep learning model.Next, we constructed 2 input, 1 output and 2 patterns neural model as in Figure 15.In this circuit, each pattern needs each circuit.For example, in the case there are 5 kinds of learning patterns, we have to construct 5 input unit circuits.However, learning time is very short."V-F" means the V-F converter circuit.The output of the subtract circuit is converted to a frequency signal by a voltagefrequency convert circuit in Fig. 15.16 means the expand network of Fig. 15.Although this model needs many neural connections, the learning speed is very high because plural data patterns learning same time and working analog real time system not depending on clock frequency.And after learning, each new connecting weight between the input layer and middle layer is picked up, it is parted potion including the connecting weights between input layer and middle layer as well as the layers of input and middle.It means stacked auto encoder process and suggest the possibility of design of many layers deep learinng model [14].To fix the connecting weights after learning process, we proposed the two-stage learning process.In the learning stage, connecting weighs are able to change depending on the teaching signal.After learning process finished, we used the sample hold circuit to fixed the connecting weights.In this situation, this circuits receives the input signal and outputs the output signal in the environment that all the connecting weights are fixed.

Conclusion
At first, we designed analog neural circuit using multiple circuits.We confirmed the operation of this network by SPICE simulation.Next, we constructed basic analog neural network by alternative current operation circuit.The input signal and connecting weight generate the Alternative current by the Amplifier circuit.Two Alternative currents are added by an additional circuit [15][16].
The frequency signal is generated by Voltage-Frequency converter circuit.The input signal of V-F converter is rectified direct current.The input of rectified circuit is the error correction signal by alternative current.The connecting weight can be changed by error-correction signal and the input frequency is depending on the output Voltage-Frequency converter circuit in the feedback learning process.This model has extremely high flexibility characteristics.It is the AC feedback Circuit for BP learning process, after getting DC current by rectifier circuit, we have to Convert from DC Voltage to AC current with Frequency.Moreover, a deep learning model has been proposed recently and developed in many applications such as image recognition and artificial intelligence.In the future, this hardware learning ng system is expected in the field of robotics, conversation system and the artificial intelligence.

Figure
Figure 2: Gilbert multiple circuits
voltage operated excellently.One neuron is composed by connecting weights, summation and threshold function.The product of input signal and connecting weights is realised by multiple circuits.summation and threshold function are realised by additional and difference circuits.

Figure 6 :
Figure 6: The convergence output of perceptron.

Figure 7 :
Figure 7: The diagram of neural circuits with threshold.

Figure 8 :
Figure 8: The learning feedback neural circuit.

Figure 10 :
Figure 10: The output Rms Value of neural circuit.

Figure 11 :
Figure 11: The output behaviour of AC operation neural circuit.

Figure 12 :
Figure 12: The convergence result of learning experiment.

Figure 10
Figure10is the output of RMS value of AC voltage by the neural circuit.In this network, two Alternative currents are added by an additional circuit.The output of the additional circuit is a modulated wave.Figure10shows the RMS value of the modulated wave.It operates satisfactorily because the output voltage increases monotonically in the general-purpose frequency range.Figure11is the output of the neural circuit.It is shown by two dimensional graph.We recognized the RMS value of output voltage is appropriate value in the two-dimensional area.

Figure 13 :
Figure 13: Basic AC operation learning neural network model.

Figure 14 :
Figure 14: The simulation results of AC feed-back neural model.

Figure 15 :
Figure 15: The structure of 2-patterns analog AC operation neural network with V-F conversion circuit.

Figure 16 :
Figure 16: The structure of enhanced analog neural network to three.layers of Fig. 14.
circuit, we have to Convert from DC Voltage to AC current with Frequency.I1 and I2 are input units.Two I1 mean two inputs.T1 and T2 means two teaching signals.W11 and W12 are connecting weights.Figure The output of the subtract circuit is converted to a frequency signal by a voltage-frequency convert circuit.It means that in the AC feedback Circuit for BP learning process, after getting DC current by rectifier