date: 2024-12-24
title: ML-Regression
status: DONE
author:
  - AllenYGY
tags:
  - NOTE
  - Regression
  - ML
publish: True

ML-Regression

Linear Regression is used for regression tasks to predict continuous outputs by minimizing the mean squared error.
Ridge Regression is a regularized version of linear regression that adds a penalty term to the loss function to prevent overfitting.
Logistic Regression is used for binary classification tasks to predict probabilities and classify data points.

Linear Regression

1. Hypothesis

Linear regression models the output as a linear function of the input features :

2. Cost Function

The cost function used is the Mean Squared Error (MSE):

3. Optimization

Gradient Descent:
Normal Equation:

Logistic Regression

1. Hypothesis

Logistic regression maps the linear function to a probability using the sigmoid function:

Prediction Rule:
- Predict if .
- Predict if .

2. Cost Function

The cost function for logistic regression is:

3. Optimization

Gradient Descent:

4. Sigmoid Function Properties

maps inputs to , representing probabilities.
Derivative:

Ridge Regression

岭回归是一种线性回归的正则化变体，通过在损失函数中加入正则化项来解决 多重共线性 和 过拟合 问题。

Loss function

岭回归的目标是最小化以下损失函数：

: 正则化参数，控制正则化强度。
- : 等价于普通的线性回归。
- : 所有。
: 正则化项，用于约束参数大小，防止过拟合。

岭回归有一个闭式解，可以通过修改普通线性回归的正规方程得到：

: 设计矩阵。
: 单位矩阵。

梯度下降解
通过梯度下降最小化损失函数：

正则化部分是对的惩罚。

Logistic Regression

Suppose predict

"" if
"" if

Cost Function and gradient

The cost function for logistic regression is defined as:

When :

if .
As , .
When , the model predicts , but .

When :

if
goes to infinite if

Simplification of Logistic Regression Cost Function

The overall cost function for logistic regression is defined as:

Cost Function:

Note: is always 0 or 1.

To fit parameters :

To make a prediction given new :

Output

The cost function for logistic regression is:

Gradient Descent Algorithm:

Repeat:

(Simultaneously update all ).

Derivative: