In cases of ill-conditioned estimation problems, not all parameters can be estimated accurately and a selection of parameter subset composed of more influential and less correlated parameters may be needed to increase parameter estimability. This paper proposes to choose a subset for estimation from transformed parameters along the directions of the principal components of the parameter covariance matrix while retaining the initial guesses for the unselected transformed parameters. Since estimating a subset of the transformed parameters can adjust all the original parameter values and any constraint on the original parameter values can still be applied to the transformed parameters, the proposed regularization method can overcome the limitation of the existing methods. i.e., parameter subset selection and truncated singular value decomposition (also known as 'principal component regression'). It is demonstrated that the proposed method can provide better parameter estimates with smaller variances than the existing parameter subset selection methods, first through statistical analysis, and then through case studies of linear and nonlinear regressions. In addition, based on the derived statistical properties, a criterion is suggested for selecting an optimal subset, which gives the smallest mean squared error of the estimates. Furthermore, with the advantage of a lower variance of the estimates, the proposed regularization method gives a more consistent choice of the number of parameters to estimate with the smallest mean squared error of the estimates even when significant errors in the initial guesses and high levels of measurement noise exist. (C) 2019 Elsevier Ltd. All rights reserved.