Degenerated simplex search method to optimize neural network error function

Abstract

Purpose

The purpose of this paper is to present a degenerated simplex search method to optimize neural network error function. By repeatedly reflecting and expanding a simplex, the centroid property of the simplex changes the location of the simplex vertices. The proposed algorithm selects the location of the centroid of a simplex as the possible minimum point of an artificial neural network (ANN) error function. The algorithm continually changes the shape of the simplex to move multiple directions in error function space. Each movement of the simplex in search space generates local minimum. Simulating the simplex geometry, the algorithm generates random vertices to train ANN error function. It is easy to solve problems in lower dimension. The algorithm is reliable and locates minimum function value at the early stage of training. It is appropriate for classification, forecasting and optimization problems.

Design/methodology/approach

Adding more neurons in ANN structure, the terrain of the error function becomes complex and the Hessian matrix of the error function tends to be positive semi‐definite. As a result, derivative based training method faces convergence difficulty. If the error function contains several local minimum or if the error surface is almost flat, then the algorithm faces convergence difficulty. The proposed algorithm is an alternate method in such case. This paper presents a non‐degenerate simplex training algorithm. It improves convergence by maintaining irregular shape of the simplex geometry during degenerated stage. A randomized simplex geometry is introduced to maintain irregular contour of a degenerated simplex during training.

Findings

Simulation results show that the new search is efficient and improves the function convergence. Classification and statistical time series problems in higher dimensions are solved. Experimental results show that the new algorithm (degenerated simplex algorithm, DSA) works better than the random simplex algorithm (RSM) and back propagation training method (BPM). Experimental results confirm algorithm's robust performance.

Research limitations/implications

The algorithm is expected to face convergence complexity for optimization problems in higher dimensions. Good quality suboptimal solution is available at the early stage of training and the locally optimized function value is not far off the global optimal solution, determined by the algorithm.

Practical implications

Traditional simplex faces convergence difficulty to train ANN error function since during training simplex can't maintain irregular shape to avoid degeneracy. Simplex size becomes extremely small. Hence convergence difficulty is common. Steps are taken to redefine simplex so that the algorithm avoids the local minimum. The proposed ANN training method is derivative free. There is no demand for first order or second order derivative information hence making it simple to train ANN error function.

Originality/value

The algorithm optimizes ANN error function, when the Hessian matrix of error function is ill conditioned. Since no derivative information is necessary, the algorithm is appealing for instances where it is hard to find derivative information. It is robust and is considered a benchmark algorithm for unknown optimization problems.

Citation

Ahmed, S. (2013), "Degenerated simplex search method to optimize neural network error function", Kybernetes, Vol. 42 No. 1, pp. 106-124. https://doi.org/10.1108/03684921311295510