We study the behaviour in zero of the derivatives of the cost function used when training non-linear neural networks. It is shown that a fair number offirst, second and higher order derivatives vanish in zero, validating the belief that 0 is a peculiar and potentially harmful location. These calculations arerelated to practical and theoretical aspects of neural networks training.
|Journal||Neural Processing Letters|
|Publication status||Published - 1998|