Design and regularization of neural networks: the optimal use of a validation set

Jan Larsen; Lars Kai Hansen; Claus Svarer; Børje Ola Mattias Ohlsson

doi:10.1109/NNSP.1996.548336

Design and regularization of neural networks: the optimal use of a validation set

Jan Larsen
, Lars Kai Hansen
, Claus Svarer
, Børje Ola Mattias Ohlsson

Copenhagen University Hospital Herlev and Gentofte

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

2124 Downloads (Orbit)

Abstract

We derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularisation parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based optimal brain damage/surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate the viability of the methods

Original language	English
Title of host publication	Proceedings of the 1996 IEEE Signal Processing Society Workshop
Publisher	IEEE
Publication date	1996
Pages	62-71
ISBN (Print)	07-80-33550-3
DOIs	https://doi.org/10.1109/NNSP.1996.548336
Publication status	Published - 1996
Event	1996 IEEE Workshop on Neural Networks for Signal Processing VI - Kyoto, Japan Duration: 4 Sept 1996 → 6 Sept 1996 Conference number: 6 http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=3974

Workshop

Workshop	1996 IEEE Workshop on Neural Networks for Signal Processing VI
Number	6
Country/Territory	Japan
City	Kyoto
Period	04/09/1996 → 06/09/1996
Internet address	http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=3974

Bibliographical note

Copyright: 1996 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE

Access to Document

10.1109/NNSP.1996.548336

SvarerFinal published version, 583 KB

OpenUrl availability

Check availability in DTU Findit

Cite this

@inproceedings{4c2dc90a15e8457a82492fd4586cf479,

title = "Design and regularization of neural networks: the optimal use of a validation set",

abstract = "We derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularisation parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based optimal brain damage/surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate the viability of the methods",

author = "Jan Larsen and Hansen, \{Lars Kai\} and Claus Svarer and Ohlsson, \{B{\o}rje Ola Mattias\}",

note = "Copyright: 1996 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE; 1996 IEEE Workshop on Neural Networks for Signal Processing VI, NNSP '96 ; Conference date: 04-09-1996 Through 06-09-1996",

year = "1996",

doi = "10.1109/NNSP.1996.548336",

language = "English",

isbn = "07-80-33550-3",

pages = "62--71",

booktitle = "Proceedings of the 1996 IEEE Signal Processing Society Workshop",

publisher = "IEEE",

address = "United States",

url = "http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=3974",

}

TY - GEN

T1 - Design and regularization of neural networks: the optimal use of a validation set

AU - Larsen, Jan

AU - Hansen, Lars Kai

AU - Svarer, Claus

AU - Ohlsson, Børje Ola Mattias

N1 - Conference code: 6

PY - 1996

Y1 - 1996

N2 - We derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularisation parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based optimal brain damage/surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate the viability of the methods

AB - We derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularisation parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based optimal brain damage/surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate the viability of the methods

U2 - 10.1109/NNSP.1996.548336

DO - 10.1109/NNSP.1996.548336

M3 - Article in proceedings

SN - 07-80-33550-3

SP - 62

EP - 71

BT - Proceedings of the 1996 IEEE Signal Processing Society Workshop

PB - IEEE

T2 - 1996 IEEE Workshop on Neural Networks for Signal Processing VI

Y2 - 4 September 1996 through 6 September 1996

ER -

Design and regularization of neural networks: the optimal use of a validation set

Abstract

Workshop

Bibliographical note

Access to Document

OpenUrl availability

Fingerprint

Cite this