1. Implement a linear model¶

• return the weight parameters w = (w1, w2, ... , wP) and the intercept parameter w0 separately where:

$$\hat{y}(\vec{w}, \vec{x}) = w_0 + w_1 x_1 + ... + w_p x_p$$

• check your returned coefficients with the built in LinearRegression class from the sklearn library, they should be within tolerance 1e-6to each other

• use a generated regression dataset from sklearn.dataset import make_regression API with parameters n_samples=1000 and n_features=20

2. Use of real data¶

• download the Communities and Crime Data Set from UCI, the task includes understanding the dataset: naming the appropiate data fields, handling missing values, etc.

• split the data in training/test sets and fit a LinearRegression model with 5-fold cross-validation on top of it - compare training and testing scores (R^2 by default) for the different CV splits, print the mean score and its standard deviation

• fit the best Lasso regression model with 5-fold grid search cross validation (GridSearchCV) on the parameters: alpha, normalize, max_iter and show the best parameters

3. Shrinkage¶

• interpret Lasso models based on its descriptive parameters by the shrinkage method described during the lecture (make a plot and check the names of the features that are not eliminated by the penalty parameter) on the data we have here [ this is an explanatory data analysis problem, try to be creative ]

• fit Ridge models and apply the shrinkage method as well, did you get what you expect?

• do you think normalization is needed here? if so, do not forget to use it in the next tasks

4. Subset selection¶

• Split the data to training and test sets and do recursice feature elimination until 10 remaining predictors with 5-fold cross-validated regressors (RidgeCV, LassoCV, ElasticNetCV) on the training set, plot their names and look up some of their meanings [ recursive feature elimination is part of sklearn but you can do it with a for loop if you whish ]

• Do all models provide the same descriptors? Check their performance on the test set! Plot all model predictions compared to the y_test on 3 different plots, which model seems to be the best?

5. ElasticNet penalty surface¶

• visualize the surface of the $objective(\alpha, \beta)$ parameters corresponding to the L1 and L2 regularizations. Select the best possible combination of the hyper-parameters that minimize the objective (clue: from scipy.optimize import minimize)

  • this task is similar to what you've seen during class, just not for MSE vs. single penalty parameter but MSE vs. two penalty parameters $\alpha, \beta$

• interpret the overall results, do you think regularization is necessary at all? do you think linear models are powerful enough on this dataset?

1.¶

• return the weight parameters w = (w1, w2, ... , wP) and the intercept parameter w0 separately where:

$$\hat{y}(\vec{w}, \vec{x}) = w_0 + w_1 x_1 + ... + w_p x_p$$

• check your returned coefficients with the built in LinearRegression class from the sklearn library, they should be within tolerance 1e-6to each other

• use a generated regression dataset from sklearn.dataset import make_regression API with parameters n_samples=1000 and n_features=20

2.¶

• download the Communities and Crime Data Set from UCI, the task includes understanding the dataset: naming the appropiate data fields, handling missing values, etc.

• split the data in training/test sets and fit a LinearRegression model with 5-fold cross-validation on top of it - compare training and testing scores (R^2 by default) for the different CV splits, print the mean score and its standard deviation

• fit the best Lasso regression model with 5-fold grid search cross validation (GridSearchCV) on the parameters: alpha, normalize, max_iter and show the best parameters

The dataset itself is loaded from the URL provided but the descriptors are still needed. Here I load the descriptors via the same method and search for the @attribute <att-name> <att-type> substring and name the columns based on the results. You obviously could have done this part manually as well.

There are basically columns with loads of missing values. It seems reasonable to drop those columns. I set a threshold on the .dropna method for half the length of the dataset and checked all the remainin rows with still remaining NaN values.

Only one NaN row remained so I filled its missing value with the dataset mean, moving on to the data types all columns except one are numeric. That one contains community names and almost all rows contain different names (~1.8k out of ~1.9k rows) so I'll drop that column as well since one-hot encoding would not provide any additional information.

Also dropping the fold columns since it contains no predictive value, and could be used for automatic cross validation.

With cross-validation for linear regression we can check how varying scores are for training and testing based on splits.

3.¶

• interpret Lasso models based on its descriptive parameters by the shrinkage method described during the lecture (make a plot and check the names of the features that are not eliminated by the penalty parameter) on the data we have here [this is an explanatory data analysis problem, try to be creative]

• fit Ridge models and apply the shrinkage method as well, did you get what you expect?

Here I chose alpha=0.00628 where 8 coefficients are none zero. Their names are visualized on the plot. Per capity white people, per capita illegal activity, etc. seem to be good descriptors of the predicted value.

Doing the same for ElasticNet [ EXTRA ]¶

4.¶

• Split the data to training and test sets and do recursice feature elimination until 10 remaining predictors with 5-fold cross-validated regressors (RidgeCV, LassoCV, ElasticNetCV) on the training set, plot their names and look up some of their meanings [recursive feature elimination is part of sklearn but you can do it with a for loop if you whish]

• Do all models provide the same descriptors? Check their performance on the test set! Plot all model predictions compared to the y_test on 3 different plots, which model seems to be the best?

To me it seems that the Lasso model is the best and not by sheer scores but based on the fact that the Ridge model got a worse score at the same time ElasticNet scores high, therefore probably the norm penalty for Lasso is more useful for this dataset than that of the Ridge model, basically you would get this result in the next excersice.

5.¶

• visualize the surface of the $objective(\alpha, \beta)$ parameters corresponding to the L1 and L2 regularizations. Select the best possible combination of the hyper-parameters that minimize the objective (clue: from scipy.optimize import minimize)
  • this task is similar to what you've seen during class, just not for MSE vs. single penalty parameter but MSE vs. two penalty parameters $\alpha, \beta$