File Download
Links for fulltext
(May Require Subscription)
- Publisher Website: 10.1109/TPAMI.2021.3068154
- Scopus: eid_2-s2.0-85103253583
- WOS: WOS:000714203900026
- Find via
Supplementary
- Citations:
- Appears in Collections:
Article: Learning Rates for Stochastic Gradient Descent with Nonconvex Objectives
Title | Learning Rates for Stochastic Gradient Descent with Nonconvex Objectives |
---|---|
Authors | |
Keywords | early stopping learning rates nonconvex optimization Stochastic gradient descent |
Issue Date | 21-Mar-2021 |
Publisher | Institute of Electrical and Electronics Engineers |
Citation | IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, v. 43, n. 12, p. 4505-4511 How to Cite? |
Abstract | Stochastic gradient descent (SGD) has become the method of choice for training highly complex and nonconvex models since it can not only recover good solutions to minimize training errors but also generalize well. Computational and statistical properties are separately studied to understand the behavior of SGD in the literature. However, there is a lacking study to jointly consider the computational and statistical properties in a nonconvex learning setting. In this paper, we develop novel learning rates of SGD for nonconvex learning by presenting high-probability bounds for both computational and statistical errors. We show that the complexity of SGD iterates grows in a controllable manner with respect to the iteration number, which sheds insights on how an implicit regularization can be achieved by tuning the number of passes to balance the computational and statistical errors. As a byproduct, we also slightly refine the existing studies on the uniform convergence of gradients by showing its connection to Rademacher chaos complexities. |
Persistent Identifier | http://hdl.handle.net/10722/329132 |
ISSN | 2023 Impact Factor: 20.8 2023 SCImago Journal Rankings: 6.158 |
ISI Accession Number ID |
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lei, Yunwen | - |
dc.contributor.author | Tang, Ke | - |
dc.date.accessioned | 2023-08-05T07:55:32Z | - |
dc.date.available | 2023-08-05T07:55:32Z | - |
dc.date.issued | 2021-03-21 | - |
dc.identifier.citation | IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, v. 43, n. 12, p. 4505-4511 | - |
dc.identifier.issn | 0162-8828 | - |
dc.identifier.uri | http://hdl.handle.net/10722/329132 | - |
dc.description.abstract | <p>Stochastic gradient descent (SGD) has become the method of choice for training highly complex and nonconvex models since it can not only recover good solutions to minimize training errors but also generalize well. Computational and statistical properties are separately studied to understand the behavior of SGD in the literature. However, there is a lacking study to jointly consider the computational and statistical properties in a nonconvex learning setting. In this paper, we develop novel learning rates of SGD for nonconvex learning by presenting high-probability bounds for both computational and statistical errors. We show that the complexity of SGD iterates grows in a controllable manner with respect to the iteration number, which sheds insights on how an implicit regularization can be achieved by tuning the number of passes to balance the computational and statistical errors. As a byproduct, we also slightly refine the existing studies on the uniform convergence of gradients by showing its connection to Rademacher chaos complexities.<br></p> | - |
dc.language | eng | - |
dc.publisher | Institute of Electrical and Electronics Engineers | - |
dc.relation.ispartof | IEEE Transactions on Pattern Analysis and Machine Intelligence | - |
dc.rights | This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. | - |
dc.subject | early stopping | - |
dc.subject | learning rates | - |
dc.subject | nonconvex optimization | - |
dc.subject | Stochastic gradient descent | - |
dc.title | Learning Rates for Stochastic Gradient Descent with Nonconvex Objectives | - |
dc.type | Article | - |
dc.description.nature | published_or_final_version | - |
dc.identifier.doi | 10.1109/TPAMI.2021.3068154 | - |
dc.identifier.scopus | eid_2-s2.0-85103253583 | - |
dc.identifier.volume | 43 | - |
dc.identifier.issue | 12 | - |
dc.identifier.spage | 4505 | - |
dc.identifier.epage | 4511 | - |
dc.identifier.eissn | 1939-3539 | - |
dc.identifier.isi | WOS:000714203900026 | - |
dc.identifier.issnl | 0162-8828 | - |