Adaptive regularization

Lars Kai Hansen; Carl Edward Rasmussen; C. Svarer; Jan Larsen

doi:10.1109/NNSP.1994.366061

Adaptive regularization

Lars Kai Hansen
, Carl Edward Rasmussen
, C. Svarer
, Jan Larsen

Technical University of Denmark

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

837 Downloads (Orbit)

Abstract

Regularization, e.g., in the form of weight decay, is important for training and optimization of neural network architectures. In this work the authors provide a tool based on asymptotic sampling theory, for iterative estimation of weight decay parameters. The basic idea is to do a gradient descent in the estimated generalization error with respect to the regularization parameters. The scheme is implemented in the authors' Designer Net framework for network training and pruning, i.e., is based on the diagonal Hessian approximation. The scheme does not require essential computational overhead in addition to what is needed for training and pruning. The viability of the approach is demonstrated in an experiment concerning prediction of the chaotic Mackey-Glass series. The authors find that the optimized weight decays are relatively large for densely connected networks in the initial pruning phase, while they decrease as pruning proceeds

Original language	English
Title of host publication	Proceedings of the 4th IEEE Workshop Neural Networks for Signal Processing
Publisher	IEEE
Publication date	1994
Pages	78-87
ISBN (Print)	07-80-32026-3
DOIs	https://doi.org/10.1109/NNSP.1994.366061
Publication status	Published - 1994
Event	1994 IEEE Workshop on Neural Networks for Signal Processing - Ermoino, Greece Duration: 6 Sept 1994 → 8 Sept 1994 Conference number: 4 https://ieeexplore.ieee.org/xpl/conhome/2959/proceeding http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=2959

Conference

Conference	1994 IEEE Workshop on Neural Networks for Signal Processing
Number	4
Country/Territory	Greece
City	Ermoino
Period	06/09/1994 → 08/09/1994
Internet address	https://ieeexplore.ieee.org/xpl/conhome/2959/proceeding http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=2959

Bibliographical note

Copyright: 1994 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE

Access to Document

10.1109/NNSP.1994.366061

CarlFinal published version, 454 KB

OpenUrl availability

Check availability in DTU Findit

Cite this

@inproceedings{0a7014b303c64d298c1def4564350b71,

title = "Adaptive regularization",

abstract = "Regularization, e.g., in the form of weight decay, is important for training and optimization of neural network architectures. In this work the authors provide a tool based on asymptotic sampling theory, for iterative estimation of weight decay parameters. The basic idea is to do a gradient descent in the estimated generalization error with respect to the regularization parameters. The scheme is implemented in the authors' Designer Net framework for network training and pruning, i.e., is based on the diagonal Hessian approximation. The scheme does not require essential computational overhead in addition to what is needed for training and pruning. The viability of the approach is demonstrated in an experiment concerning prediction of the chaotic Mackey-Glass series. The authors find that the optimized weight decays are relatively large for densely connected networks in the initial pruning phase, while they decrease as pruning proceeds",

author = "Hansen, \{Lars Kai\} and Rasmussen, \{Carl Edward\} and C. Svarer and Jan Larsen",

note = "Copyright: 1994 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE; 1994 IEEE Workshop on Neural Networks for Signal Processing, NNSP 1994 ; Conference date: 06-09-1994 Through 08-09-1994",

year = "1994",

doi = "10.1109/NNSP.1994.366061",

language = "English",

isbn = "07-80-32026-3",

pages = "78--87",

booktitle = "Proceedings of the 4th IEEE Workshop Neural Networks for Signal Processing",

publisher = "IEEE",

address = "United States",

url = "https://ieeexplore.ieee.org/xpl/conhome/2959/proceeding, http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=2959",

}

TY - GEN

T1 - Adaptive regularization

AU - Hansen, Lars Kai

AU - Rasmussen, Carl Edward

AU - Svarer, C.

AU - Larsen, Jan

N1 - Conference code: 4

PY - 1994

Y1 - 1994

N2 - Regularization, e.g., in the form of weight decay, is important for training and optimization of neural network architectures. In this work the authors provide a tool based on asymptotic sampling theory, for iterative estimation of weight decay parameters. The basic idea is to do a gradient descent in the estimated generalization error with respect to the regularization parameters. The scheme is implemented in the authors' Designer Net framework for network training and pruning, i.e., is based on the diagonal Hessian approximation. The scheme does not require essential computational overhead in addition to what is needed for training and pruning. The viability of the approach is demonstrated in an experiment concerning prediction of the chaotic Mackey-Glass series. The authors find that the optimized weight decays are relatively large for densely connected networks in the initial pruning phase, while they decrease as pruning proceeds

AB - Regularization, e.g., in the form of weight decay, is important for training and optimization of neural network architectures. In this work the authors provide a tool based on asymptotic sampling theory, for iterative estimation of weight decay parameters. The basic idea is to do a gradient descent in the estimated generalization error with respect to the regularization parameters. The scheme is implemented in the authors' Designer Net framework for network training and pruning, i.e., is based on the diagonal Hessian approximation. The scheme does not require essential computational overhead in addition to what is needed for training and pruning. The viability of the approach is demonstrated in an experiment concerning prediction of the chaotic Mackey-Glass series. The authors find that the optimized weight decays are relatively large for densely connected networks in the initial pruning phase, while they decrease as pruning proceeds

U2 - 10.1109/NNSP.1994.366061

DO - 10.1109/NNSP.1994.366061

M3 - Article in proceedings

SN - 07-80-32026-3

SP - 78

EP - 87

BT - Proceedings of the 4th IEEE Workshop Neural Networks for Signal Processing

PB - IEEE

T2 - 1994 IEEE Workshop on Neural Networks for Signal Processing

Y2 - 6 September 1994 through 8 September 1994

ER -

Adaptive regularization

Abstract

Conference

Bibliographical note

Access to Document

OpenUrl availability

Fingerprint

Cite this