Parametric Flatten-T Swish: An Adaptive Nonlinear Activation Function for Deep Learning

Authors

  • Hock Hung Chieng Faculty of Information Technology and Computer Science, Universiti Tun Hussein Onn Malaysia, Malaysia
  • Noorhaniza Wahid Faculty of Information Technology and Computer Science, Universiti Tun Hussein Onn Malaysia, Malaysia
  • Pauline Ong Faculty of Mechanical and Manufacturing Engineering, Universiti Tun Hussein Onn Malaysia, Malaysia

DOI:

https://doi.org/10.32890/jict2021.20.1.2

Keywords:

Activation function, deep learning, Flatten-T Swish, non-linearity, ReLU

Abstract

QActivation function is a key component in deep learning that performs non-linear mappings between the inputs and outputs. Rectified Linear Unit (ReLU) has been the most popular activation function across the deep learning community. However, ReLU contains several shortcomings that can result in inefficient training of the deep neural networks, these are: 1) the negative cancellation property of ReLU tends to treat negative inputs as unimportant information for the learning, resulting in performance degradation; 2) the inherent predefined nature of ReLU is unlikely to promote additional flexibility, expressivity, and robustness to the networks; 3) the mean activation of ReLU is highly positive and leads to bias shift effect in network layers; and 4) the multilinear structure of ReLU restricts the non-linear approximation power of the networks. To tackle these shortcomings, this paper introduced Parametric Flatten-T Swish (PFTS) as an alternative to ReLU. By taking ReLU as a baseline method, the experiments showed that PFTS improved classification accuracy on SVHN dataset by 0.31%, 0.98%, 2.16%, 17.72%, 1.35%, 0.97%, 39.99%, and 71.83% on DNN-3A, DNN-3B, DNN-4, DNN-5A, DNN-5B, DNN-5C, DNN-6, and DNN-7, respectively. Besides, PFTS also achieved the highest mean rank among the comparison methods. The proposed PFTS manifested higher non-linear approximation power during training and thereby improved the predictive performance of the networks.

References

Agostinelli, F., Hoffman, M., Sadowski, P., & Baldi, P. (2015). Learning activation functions to improve deep neural networks. Workshop Track Proceedings of the 3rd International Conference on Learning Representations (ICLR). CoRR. https://arxiv.org/abs/1412.6830

Alcantara, G. (2017). Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks. CoRR. https://arxiv.org/ abs/1710.11272

Chen, J., Chen, J., Zhang, R., & Hu, X. (2019). Toward a Brain-Inspired System: Deep Recurrent Reinforcement Learning for a Simulated Self-Driving Agent. Frontiers in neurorobotics, 13(40) https://doi.org/10.3389/fnbot.2019.00040

Chieng, H. H., Wahid, N., Ong, P., & Perla, S. R. K. (2018). Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning. International Journal of Advances in Intelligent Informatics, 4(2), 76-86. https://doi.org/10.26555/ijain.v4i2.249

Ciuparu, A., Nagy-Dăbâcan, A., & Mureşan, R. C. (2019). Soft++, a multi-parametric non-saturating non-linearity that improves convergence in deep neural architectures. Neurocomputing, 384, 376-388. https://doi.org/10.1016/j.neucom.2019.12.014

Clevert, D. A., Unterthiner, T., & Hochreiter, S. (2016). Fast and accurate deep network learning by exponential linear units (ELUs). Proceedings of the International Conference on Learning Representations (ICLR), 1-15. https://arxiv.org/abs/1511.07289

Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research, 9, 249-256. https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017). Neuroscience-inspired artificial intelligence. Neuron, 95(2), 245-258. https://doi.org/10.1016/j.neuron.2017.06.011

He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, 1026-1034. https://doi:10.1109/ICCV.2015.123 Journal of ICT, 20, No. 1 (January) 2021, pp: 21-

Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In 32nd International Conference on Machine Learning (ICML), 37, 448-456. https://dl.acm.org/doi/10.5555/3045118.3045167

Jagtap, A. D., Kawaguchi, K., Karniadakis, G. E. (2019). Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404, 109136. https://doi: 10.1016/j.jcp.2019.109136

Jinsakul, N., Tsai, C. F., Tsai, C. E., & Wu, P. (2019). Enhancement of Deep Learning in Image Classification Performance Using Xception with the Swish Activation Function for Colorectal Polyp Preliminary Screening. Mathematics, 7(12), 1170. https://doi:10.3390/ MATH7121170

Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). Self-normalizing neural networks. Advances in Neural Information Processing Systems, 972-981. https://dl.acm.org/doi/10.5555/3294771.3294864

Laurent, T., & Von Brecht, J. H. (2018). The multilinear structure of ReLU networks. In 35th International Conference on Machine Learning (ICML), 80, 2908-2916. http://proceedings.mlr.press/v80/laurent18b. html

Lin, G., & Shen, W. (2018). Research on convolutional neural network based on improved Relu piecewise activation function. Procedia Computer Science, 131, 977-984. http://doi.org/10.1016/j.procs.2018.04.239

Liu, Y., Zhang, J., Gao, C., Qu, J., & Ji, L. (2019). Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks. In 2019 IEEE 5th International Conference on Computer and Communications (ICCC), 2000-2008. https://doi.org/10.1109/ICCC47050.2019.9064398

Maas, A. L., Hannun, A. Y., & Ng, A. Y. (2013). Rectifier nonlinearities improve neural network acoustic models. Proceedings of the 30th International Conference on Machine Learning (ICML), Workshop on Deep Learning for Audio, Speech, and Language Processing, 30(1), 3. https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf

Mohamed, A. R., Dahl, G. E., & Hinton, G. (2011). Acoustic modeling using deep belief networks. IEEE transactions on audio, speech, and language processing, 20(1), 14-22. https://doi.org/10.1109/TASL.2011.2109382

Nair, V., & Hinton G. E. (2010). Rectified linear units improve Restricted Boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), 807-814. https://dl.acm.org/ doi/10.5555/3104322.3104425

Ohn, I., & Kim, Y. (2019). Smooth function approximation by deep neural networks with general activation functions. Entropy, 21(7), 627. https://doi:10.3390/e21070627 Journal of ICT, 20, No. 1 (January) 2021, pp: 21-

Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017). Regularizing neural networks by penalizing confident output distributions. In Proceedings of the 5th International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1701.06548

Qian, S., Liu, H., Liu, C., Wu, S., & Wong, H. S. (2018). Adaptive activation functions in convolutional neural networks. Neurocomputing, 272, 204-212. https://doi:10.1016/j.neucom.2017.06.070

Qiu, S., Xu, X., & Cai, B. (2018). FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks, In 24th International Conference on Pattern Recognition (ICPR), 1223-1228. https://doi:10.1109/ICPR.2018.8546022

Ramachandran, P., Zoph, B., & Le, Q. V. (2018). Searching for activation functions. Workshop Track Proceedings of the 6th International Conference on Learning Representations (ICLR).

Robbins, H., & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3), 400-407. https://doi:10.1214/ aoms/1177729586

Scardapane, S., Comminiello, D., Hussain, A., & Uncini, A. (2017). Group sparse regularization for deep neural networks. Neurocomputing, 241, 81-89. https://doi.org/10.1016/j.neucom.2017.02.

Sütfeld, L. R., Brieger, F., Finger, H., Füllhase, S., & Pipa, G. (2018). Adaptive blending units: Trainable activation functions for deep neural networks. https://arxiv.org/abs/1806.10064

Tripathi, G. C., Rawat, M., & Rawat, K. (2019). Swish Activation Based Deep Neural Network Predistorter for RF-PA. In IEEE Region 10 Annual International Conference (TENCON), 1239-1242. https://doi:10.1109/ TENCON.2019.8929500

Trottier, L., Gigu, P., Chaib-Draa, B. (2017). Parametric exponential linear unit for deep convolutional neural networks. Proceedings of 16th IEEE International Conference on Machine Learning and Applications (ICMLA), 207-214. https://doi:10.1109/ICMLA.2017.00038

Wang, L., Li, Q., & Guo, H. (2019). A Research on Deep Learning Model for Face Emotion Recognition Based on Swish Activation Function. Journal of Image and Signal Processing, 8(3), 110-120. https://doi:10.12677/JISP.2019.83016

Yarotsky, D. (2018). Optimal approximation of continuous functions by very deep relu networks. Proceedings of Machine Learning Research, Vol 75, 1-11. http://proceedings.mlr.press/v75/yarotsky18a

Zhou, Z. H., & Feng, J. (2017). Deep forest: towards an alternative to deep neural networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, 3553-3559, AAAI Press. https://www.ijcai.org/Proceedings/2017/497

Downloads

Published

04-11-2020

Issue

Vol. 20 No. 1 (2021): Journal of Information and Communication Technology (JICT) Vol.20, No.1, January 2021

Section

Articles

How to Cite

Chieng, H. H., Wahid, N., & Ong, P. (2020). Parametric Flatten-T Swish: An Adaptive Nonlinear Activation Function for Deep Learning. Journal of Information and Communication Technology, 20(1), 21-39. https://doi.org/10.32890/jict2021.20.1.2

Research impact

Harvested 2026-09-06
0 citations recorded so far

Counts differ between services because each indexes a different body of literature. None of them is the whole picture.

Identifiers DOI 10.32890/jict2021.20.1.2

Most read articles by the same author(s)