File Download

There are no files associated with this item.

Supplementary

Conference Paper: The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift

TitleThe Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Authors
Issue Date12-Dec-2022
Abstract

We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional distribution of the output given the input covariates is similar across the two domains. We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data (both conducted by online SGD) for this problem. We establish sharp instance-dependent excess risk upper and lower bounds for this approach. Our bounds suggest that for a large class of linear regression instances, transfer learning with O(N2 ) source data (and scarce or no target data) is as effective as supervised learning with N target data. In addition, we show that finetuning, even with only a small amount of target data, could drastically reduce the amount of source data required by pretraining. Our theory sheds light on the effectiveness and limitation of pretraining as well as the benefits of finetuning for tackling covariate shift problems.


Persistent Identifierhttp://hdl.handle.net/10722/340330

 

DC FieldValueLanguage
dc.contributor.authorWu, Jingfeng-
dc.contributor.authorZou, Difan-
dc.contributor.authorBraverman, Vladimir-
dc.contributor.authorGu, Quanquan-
dc.contributor.authorKakade, and Sham M -
dc.date.accessioned2024-03-11T10:43:20Z-
dc.date.available2024-03-11T10:43:20Z-
dc.date.issued2022-12-12-
dc.identifier.urihttp://hdl.handle.net/10722/340330-
dc.description.abstract<p>We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional distribution of the output given the input covariates is similar across the two domains. We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data (both conducted by online SGD) for this problem. We establish sharp instance-dependent excess risk upper and lower bounds for this approach. Our bounds suggest that for a large class of linear regression instances, transfer learning with O(N2 ) source data (and scarce or no target data) is as effective as supervised learning with N target data. In addition, we show that finetuning, even with only a small amount of target data, could drastically reduce the amount of source data required by pretraining. Our theory sheds light on the effectiveness and limitation of pretraining as well as the benefits of finetuning for tackling covariate shift problems.</p>-
dc.languageeng-
dc.relation.ispartofAdvances in Neural Information Processing Systems (28/11/2022-09/12/2022, New Orleans)-
dc.titleThe Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift -
dc.typeConference_Paper-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats