Minimizing Classification Energy of Binarized Neural Network Inference for Wearable Devices

In recent years, the demand for using wearable technology has increased dramatically with the advancement of technology and the growth of the Internet of Things (IoT). Wearable devices are extending their applications in many diverse domains from consumer electronics, such as smartwatches and activity trackers for monitoring one’s fitness and wellness, to medical devices that track a patient’s vital physiological signs such as heart rate and brain activity. Such devices usually process real-time data, read from multimodal sensors continuously, and suffer from resource-bound and limited battery budget due to their small size, online monitoring, and portability. Therefore, minimizing the power dissipation of these devices while meeting real-time requirements is a subject of interest [1, 2, 3, 4, 5].

Deep Neural Networks (DNNs) are among the effective machine learning (ML) approaches that are employed majorly in the processing units of such devices. DNNs can extract featured information from raw multimodal time-series signals without much prior knowledge of the signal [2]. However, DNNs with high-precision weights consume high-energy operations and require large blocks of memories for storing and processing the DNN model. To tackle the problem of memory and power consumption, weight pruning and weight quantization techniques [6, 7, 8] as well as low precision weight networks [9, 10, 11] have been proposed in recent years as effective approaches to compressing DNN models. [9], [10], and [11] proposed respectively BinaryConnect (BC), Binarized Neural Network (BNN), and Binary-Weight-Networks (BWN), all of which constrain every weight of the neural network to either $-1$ or $+1$, that can be stored in the smallest possible unit of memory, i.e. 1 bit ($0$ for $-1$ and $1$ for $+1$), thereby allowing to pack as many weights in as a memory entry can accommodate. In BNNs, not only the weights but the feature-map is also constrained to $-1$ or $+1$, thus resulting in bit-wise operations between each neuron output and every synapse weight. Due to the minimal aspects of binary weight DNNs, they have gained significant attention recently. BNNs work well on small datasets [9], and will be shown in this work that they can be employed on physiological time-series data as well.

Time-series data classification in ML applications is conventionally processed with techniques such as Dynamic Time Warping (DWT), Recurrent Neural Networks (RNN), and Long Short-Term Memory (LSTM) [12]. These techniques have complicated algorithms and hardware implementation, and mainly deal with complex input data such as text or speech. DNNs, nevertheless, have also been used in the classification of time-series data [2, 13]. In this approach, the real-time data is cut and buffered into frames of samples that may or may not contain specific patterns, e.g. seizure pattern, that annotate the frame with specific labels. The frame of temporal data can be multi-channel and can be used as raw data or processed in the frequency domain to feed a DNN, or represented as an image-like frame whose sequence of image rows follow the sequence of sensors that pick data, e.g. EEG sensors on the skull can represent the brain activity spatially and temporally [14].

In this paper, a low-power scalable hardware is proposed for multimodal physiological data classification that uses BNNs. In its Verilog HDL, the number of PEs and memory width can be adjusted to fit the minimal energy consumption given an application. The hardware is designed to be minimal, and use resource sharing and on-chip memories at its finest.

The main contributions of this paper include:

• Train two physiological case studies including Physical Activity Monitoring, and Stress Detection for BNNs

• Improve BNN accuracy by increasing the number of parameters to achieve the same accuracy as full-precision

• Propose a scalable parallel hardware for BNN inference that can be configured for different number of PEs, and memory width, both of which adjustable for minimal classification energy given a case study.

• Propose an operation skipping technique, referred to as Pool-Skipping, for Max-Pool layer in BNNs that can skip more than 25% of effectless operations accompanied with MP and 22% of the BNN operations in total.

• Analyze hardware parameters, including energy vs memory width, and energy vs number of PEs and find the optimal configuration per case study on FPGA and on ASIC.

• Synthesis and Implementation of the platform on FPGA and post place-and-route ASIC layout in 65nm CMOS technology, and provide the energy and time reports of the two case studies.

Several multimodal data classification methods have been proposed recently. [2] proposed a complexity-reduced Deep Convolutional Neural Network (DCNN) for physiolog- ical case studies and a 16-bit hardware platform for their inference. [13] proposed an architecture that uses a CNN per variable and tested their CNN on Congestive Heart Failure Detection and on Physical Activity Monitoring datasets with detection accuracies of respectively 94% and 93%. [15] proposed a scalable system that addresses concurrent activity recognition. Their model extracts temporal and spatial features from a multimodal sensory data by means of a 7 layer CNN followed by an LSTM and is tested on three activity datasets. [16] introduced DeepSense which is a deep learning framework that integrates DNNs and RNNs for addressing feature customization challenges in sensory datasets. [17] investigated the opportunity to use deep learning for identifying nonintuitive features from cross-sensor correlations by means of an RBM. In [1] a kernel decomposing scheme in binary-weight networks is proposed that skips redundant computations and achieves 22% energy reduction on image classification. BiNMAC is proposed in [18] which is a programmable manycore accelerator for BNNs designed for physiological and Image processing case studies.

BNNs constrain both weights and the feature-map to either $-1$ or $+1$. Similar to full-precision DNNs, BNNs consist of layers equivalent to those of the standard DNNs, albeit implemented rather differently; for the first layer of BNN, data from each input channel are in full-precision values, but model weights of the BNN are trained and constrained to either $-1$ or $+1$. Therefore, Multiply-Accumulation (MAC) operations are replaced by a series of ADDs/SUBs. For the next layers, all input features are binarized using binary activation functions (AF) and packed inside registers. Therefore, a MAC operation between two vectors that hold such packed weights can be performed by summing (considering $0$ as $-1$) over the result of the bit-wise $xnor$ of the two vectors. The summation over the bits of a register is referred to as $population-count$ and denoted by $pcnt$ in this work.

In order to apply the $AF$ to the real-valued variables and transform them to either $-1$ or $+1$, deterministic binarization function is proposed for AF by [10] which is essentially a $sign$ function as shown in equation (1), where $x$ is the real-valued variable and $x_b$ is its binarized value.

Thus, the feed-forward path of a layer, that is traditionally denoted with $Y=AF(W.X+B)$ in standard DNNs, can be formulated for BNNs by equation (2):

where X and W are respectively tensors of input (feature-map) and weights with bias ($B$) implicitly included, $xnor$ performs bit-wise $xnor$ operations on rows from W and columns from X, $pcnt$ sums over the result of this bit-wise operation, and sign is the AF applied to the generated scalar value from $pcnt$. As explained earlier, from a hardware point of view it is more efficient for both binarized weights and feature-map to be packed in memory entries. For example, one row of a binarized $W_{128\times 128}$ can be stored in 4 memory entries of a 32-bit machine, and the whole matrix can be packed inside 512 entries (2KB). Therefore, practically in an M-bit computing machine, the population-count part in equation (2) should be performed as per every chunk of aligned bit-wise $xnor$. Hence, before applying the activation function the population-count needs to be carried out several times until the completion of fetching all patch elements that contribute to generating one output value. This accumulation, before applying the $sign$ AF, can be denoted by the following equation [9]:

where $W_M$ and $X_M$ are chunks of $W$ and $X$ that are sequentially read from packed memory entries of an M-bit machine until they are fetched completely. Only then one can infer $Y=sign\left(temp\right)$ from equations (2) and (3).

For the last layer, weights can either be constrained to $-1$ or $+1$, that would be treated similarly as previous layers, or trained with full-precision values that, with respect to the binarized activation values coming from the previous layer would convert the MAC operation into ADD/SUB over the weight values, analogous to the first layer. Binarizing the last layer of BNNs can sometimes result in failure and therefore different techniques have been proposed to deal with the last layer [19, 20]. We use high-precision (16-bit) weights for the last layer of the first case study in this paper because it resulted in a stabilized BNN accuracy during trainaing over 100 epochs.

Max-Pool (MP) is a frequently-employed layer in DNNs that sub-samples the feature-map, and is traditionally placed before $AF$. It can be shown that if $AF$ is a non-descending function of its input, then placing $MP$ after $AF$ results in equivalent outcome. In BNNs therefore:

Placing $MP$ after $AF$ layer in BNNs has two advantages: 1) a bulky comparator that would have to operate on the real-valued results from the previous Conv/FC layer in order to execute $MP$ before $AF$, can be converted to an $or$ gate if $MP$ follows $AF$ with binarized output. Because, if $y_i$ $\in$ $\{$$0$, $1$$\}$, then $max$($y_1$, $y_2$, …, $y_n$) = $or$($y_1$, $y_2$, …,$y_n$), where $y_i$ and $n$ are elements and the pool size of the MP 2) as soon as one of the elements of the pool, at which the $MP$ seeks the maximum, is equal to the maximum possible value ($1$ in BNNs), the pool exploring can be asserted as ‘done’, and the rest of the exploration in the pool can be skipped, that is conducive to skipping of large portion of computation imposed from the preceding Conv/FC layer. We refer to this property as Pool-Skipping and in Section VI show that it can skip up to 22% of the total operations for the first case study that has 3 $MP$ layers.

Since in Neural Networks one operation is usually considered as either one multiplication or one addition, therefore each time either a bit-wise $xnor$ or a $pcnt$ is executed, the number of carried out operations is equal to the number of vector bits to be executed. Consequently in a BNN process, by fusing and well pipelining the $pcnt$ and bit-wise $xnor$ operations in an M-bit machine, one can achieve 2$\times$M operations per execution of equation 3. This implies that the performance of an M-bit BNN processing machine can be proportional to M, which decides its registers, data bus, memory width, etc. We will evaluate this feature in Section VI.

BNN models can compress the model size by 8 to 64 times, at the expense of some accuracy loss. The accuracy loss can usually be compensated by augmenting the BNN model size by 2-11$\times$ [20]. In order to present a fair comparison between our hardware and related works, in this section, we introduce two case studies that were used in [2] and [18]. We use Torch framework from [9] to train the BNN models and to grab the weights for inference on hardware.

In this section, we introduce a minimal hardware architecture that is scalable with the number of PEs and memory width that can be adjusted to match the time and energy requirements given an application. For the parallelization scheme, we use output channel tiling as in [23] in which every PE contributes to generating an independent channel of the feature-map. This scheme is shown to result in the best computation to memory communication ratio [24]. It also facilitates implementing the Pool-Skipping method with a simple control logic.

The high-level abstraction of the hardware is depicted in Fig.3. The main components of the hardware include Filter Memory to store the model weights, an Input-Memory and a Feature-Map Memory that are used for the intermediate data and alternately switch their tasks, i.e. buffering the input data/feature-map to be exported to PEs in one memory and importing the output feature-map from the PEs in another memory. All the three mentioned memories are shared between the PEs and addressed by a global controller. The global controller is the only unit of the hardware that uses a multiplier for address calculations.

Each PE comprises a Filter cache to temporarily store an individual filter for the process, and an Output cache to concatenate the single output bits and prepare them to be sent to the Feature-Map Memory along with other PEs. Once this cache is full, all the PEs concatenate their first bit of output-cache and send the packed packet out to the feature-map memory. Then, next bits are sent out accordingly. Resource sharing is carefully taken into consideration such that the major logic of every PE is only a pipeline of bit-wise $xnor$, $pcnt$, and an accumulator to satisfy the equation (2). The accumulator ADDs/SUBs over either the input data w.r.t binarized packed filter for the first BNN layer, or on the output of $pcnt$ for mid-layers, or over the filter data w.r.t binarized packed data for the last BNN layer. Two shifters, not depicted in Fig. 3 to avoid obfuscation, shift the packed filter/data for the first/last layers. A local controller, by means of Multiplexers to allocate the existing resources, handles the data-flow such that FC and Conv layers with different strides are executed with respect to the BNN layer.

A bit-wise $or$ gate is implemented after the Output cache to perform MP and is by-passed if there is no MP. Pool-Skipping is implemented in such a way that, given MP, whenever there is a +1 in the pool-under-explore, +1 is written quickly to its relevant location in the Output cache, the pipeline is flushed, and the operation is skipped to a stride as wide as the pool size. The would-be adjacent bits in the Output cache are left as ‘don’t care’ as their old values will be masked by the effect of the written +1.

We compare our proposed hardware on the two case studies with two relevant works, [2] and [18], from which we obtained our BNN configurations. The comparison is summarized in Table IV. In summary, despite 9.5$\times$ augmentation in the number of parameters, the optimal config. for the Stress Detection is 4.5$\times$ and 6.8$\times$ more energy efficient as compared to [2] implemented respectively on Artix-7 FPGA and on ASIC 65nm. For the Physical Activity, the minimal hardware with no MP and Conv logics, is 250$\times$ more energy efficient as compared to a programmable, yet low-power manycore platform in [18].

We proposed a scalable hardware for inference of BNNs that have been trained for physiological datasets. The proposed hardware has flexibility in adjusting the memory width and number of PEs, both adjustable for the most energy efficient configuration while meeting time requirements given an application. Two case studies including Physical Activity Monitoring and Stress Detection are trained and deployed on the hardware, and for each, the optimal number of PEs and memory-width is sought. Our implementation results with the two case studies on Artix-7 FPGA and on 65nm ASIC CMOS standard cell library indicate that the configuration parameters of the hardware, if adjusted well, can optimize the energy consumption up to 4$\times$ and 2.5$\times$ respectively on FPGA and on ASIC. An operation skipping method, named Pool-Skipping, is also proposed for BNNs that skips effectless operations that precede a Max-Pool layer, and can skip up to 22$\%$ of operations in the Stress Detection case study. When compared to the related works that use the same case studies on the same target platforms, FPGA and ASIC, and with the same classification accuracy, our hardware is respectively 4.5$\times$ and 250$\times$ more energy efficient for the Stress Detection on FPGA and Physical Activity Monitoring on ASIC.