ACTIVATION FUNCTIONS IN SINGLE HIDDEN LAYER FEED-FORWARD NEURAL NETWORKS
Abstract
Abstract
Especially in the last decade, Artificial Intelligence (AI) has gained increasing popularity as the neural networks represent incredibly exciting and powerful machine learning-based techniques that can solve many real-time problems. The learning capability of such systems is directly related with the evaluation methods used. In this study, the effectiveness of the calculation parameters in a Single-Hidden Layer Feedforward Neural Networks (SLFNs) will be examined. We will present how important the selection of an activation function is in the learning stage. A lot of work is developed and presented for SLFNs up to now. Our study uses one of the most commonly known learning algorithms, which is Extreme Learning Machine (ELM). Main task of an activation function is to map the input value of a neural network to the output node with a high learning or achievement rate. However, determining the correct activation function is not as simple as thought. First we try to show the effect of the activation functions on different datasets and then we propose a method for selection process of it due to the characteristic of any dataset. The results show that this process is providing a remarkably better performance and learning rate in a sample neural network.
Keywords: Machine Learning, SLFN, ELM.
Keywords
Full Text:
PDFReferences
UCI Irvine Machine Learning Repository [online], http://archive.ics.uci.edu/ml/datasets.html , [Last visited: 02 Jan 2019],
Huang GB, Zhu QY, Siew CK, Extreme learning machine, Theory and applications, Neurocomputing 70, 2006; 489-501.
Huang GB, Ding X, Zhou H, Optimization method based extreme learning machine for classification, Neurocomputing 74, 2010; 155-163.
Feng G, Qian Z, Zhang X, Evolutionary selection extreme learning machine optimization for regression, Soft Computing 16, 2012; 1485-1491.
Huang GB, Zhou H, Ding X, Extreme learning machine for regression and multiclass classification, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42, 2012; 513-529.
ELM Classification web page [online], http://www.ntu.edu.sg/home/egbhuang/ elm_random_hidden_nodes.html [Last visited: 02 Jan 2019],
Glorot X, Bengio Y, Understanding the difficulty of training deep feedforward neural networks, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010.
Liu Q and Wang J, A One-Layer Recurrent Neural Network with a Discontinuous Hard-Limiting Activation Function for Quadratic Programming, IEEE Transactions On Neural Networks, Vol. 19, No. 4, April 2008.
Parascandolo G, Huttunen H, Virtanen T, Taming the Waves: Sine as activation function in deep neural networks, ICLR 2017 conference submission, 2017.
Kohavi R. A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Conference, International Joint Conference on Artificial Intelligence (IJCAI), 1995
Gibbs M.N., Variational Gaussian process classifiers. IEEE Transactions on Neural Networks. 11 (6), Nov 2000; 1458–1464.
Article Metrics
Metrics powered by PLOS ALM
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.