Scikit-learn Tutorial
Scikit‑learn is one of the most widely used machine learning libraries in Python. It offers clean, consistent APIs for building, training, and evaluating statistical models—from simple linear regressions to more advanced ensemble methods. This tutorial introduces several foundational models and demonstrates how to apply them to real‑world datasets.
0. Data Preparation
Let’s first load in the data and remove all missing values.
1. Linear Regression
When the outcome variable is continuous, the simpliest model is a linear regression model. Suppose we are interested in modeling Height with Weight and Age. To fit a linear regression model, we first need to import LinearRegression from sklearn.linear_model.
Then, create the predictors X and outcome y arrays from df.
Now, we are ready to fit a linear regression model. We begin by defining the model with LinearRegression, the fit_intercept option determine whether you would like to get an estimate for the intercept. Use fit(X,y) to fit the model. The first parameter is always your predictors dataframe and the second parameter is your outcome dataframe.
We can show the fitted model coefficients using .intercept_ and .coef_.
Suppose we are now interested in modeling Height with Weight and Ethnicity. When the predictors are categorical, we have to first create dummy variables (as introduced in the Pandas Tutorial).
Now, we can define the predictors and response dataframe with dummy variables and fit the model.
Similarly, we can show the fitted model coefficients using .intercept_ and .coef_.
Using score on a linear regression model will return the coefficient of determination.
As an exercise, fit a model to explain BMI using Weight and Height.
2. Logistic Regression
Suppose we are interested in modeling Stroke with Weight and LDL_Cholesterol. When the outcome variable is binary, the simplest model is a logistic regression model. To fit a logistic regression model, we first need to import LogisticRegression from sklearn.linear_model.
Encode the binary outcome variable so that they entires are no longer text. We can use map to do it. In the following code, we are mapping “Yes” to 1 and “No” to 0 when redefining the Stroke column.
Now, we can define the predictors and response dataframe and fit the model.
Show the fitted model coefficients using .intercept_ and .coef_.
As an exercise, fit a logistic regression model to predict high_LDL status using Weight and Height. Similar to the Stroke column, map “yes” to 1 and “no” to 0 when redefining the high_LDL column.
Often time, we will do a train-test split on the data to evaluate the model. This can be done through train_test_split, which is a method that can be imported from sklearn.model_selection. We need to specify the predictors X and outcome y will be split, the proportion that goes into the testing data, as well as a random seed.
Below, we are splitting the dataset as train-test in 70:30 proportion with a random seed at 228.
We can fit our model using the training datasets X_train, y_train.
With the fitted model, make prediction on the unseen data X_test using predict().
To evaluate the prediction performance, we can compute the confusion matrix and accuracy. These performance metrics can be imported from sklearn.metrics. More details can be found at sklearn metrics.
Aside from metric values, we can also generate the ROC curve and calculate area under curve (AUC). To do so, we first obtain the probability of each test observation being classified into the “yes” class using predict_proba. The returned estimates for all classes are ordered by the label of classes.
3. LASSO and Ridge Regression
When we have too many predictors, we may want to eliminate the less important ones for better model interpretation. This can be done through LASSO and Ridge regression to penalize excess predictors.
LASSO and Ridge differ only by the weight of the penalization, with ridge imposing heavier weights.
First, load all the necessary methods.
Then, define the predictors and response variable. In application, people tend to standardize the data. This can be performed using StandardScaler().
In penalized regression, we leverage cross validation to choose the best model (i.e. which predictors to retain). The general procedure involves repeatedly partitioning the available data into multiple folds (i.e. subsets), training the model on some subsets, and testing it on the remaining ones.
This code fits a range of LASSO model with varying penalty alpha and choose the best model using leave-one-out cross validation.
This code fits a range of Ridge model with varying penalty alpha and choose the best model using leave-one-out cross validation (LOOCV).
We can determine the best penality term using .alpha_ and return the mean testing squared error using mean_squared_error(y_test, model.predict(x_test)).
We can also generate a plot to show the testing MSE trend as we impose different penalties.
Note
Python and R may generate slightly different results because their random‑number systems differ. Even with the same seed, the training and testing splits will not be identical across the two languages. Hence, the confusion matrix, accuracy, best penality term, etc are different. Nonetheless, we should expect the results to be similar, if not exactly the same.
4. K-Nearest Neighbour
KNN is a non‑parametric method (so we do not estimate any coefficients, unlike regression models). For a given test input, it finds the k nearest training inputs and output a prediction that is the most common label among the neighbours. In short, it makes predictions by looking at the k nearest training samples.
First, import KNeighborsClassifier from sklearn.neighbors.
Then, define the predictors and response variable. Similarly, conduct a train-test split for model validation purpose. This time, we use a 80:20 split.
In practice, we choose k based on cross validation. In this example, let’s set it as 5.
Use .fit() and .predict() to fit the model and test it on the testing data. Model accuracy can be obtained using accuracy_score().