Auto Byte

Science AI

# 如何通过牛顿方法解决Logistic回归问题

1. 微分和链式法则（微积分）

2. 偏微分与梯度（多元微积分）

3. 基本向量运算（线性代数）

4. NumPy 的基本理解

5. 独立概率的基本理解

### 模型

logistic 函数，也就是 sigmoid 函数，能够完成所有这些事情，函数表达式如下：

### 数学：将所有的放在一起

1. 我们的对数似然函数的梯度的导数（Derivation of the Gradient of our Log-Likelihood）, 吴恩达课程笔记 17-18 页

2. 海森矩阵的求解其实相当直接，如果你曾经计算过梯度，你会在吴恩达课件笔记中「对 sigmoid 函数求导 g′(z)」那一部分看到。

ℓ(θ) 的梯度是：

ℓ(θ) 的海森矩阵是：

### 实现牛顿方法

``````def sigmoid(x, Θ_1, Θ_2):
z = (Θ_1*x + Θ_2).astype("float_")
return 1.0 / (1.0 + np.exp(-z))  ``````

``````def log_likelihood(x, y, Θ_1, Θ_2):
sigmoid_probs = sigmoid(x, Θ_1, Θ_2)
return np.sum(y * np.log(sigmoid_probs)
+ (1 - y) * np.log(1 - sigmoid_probs))
``````

``````def gradient(x, y, Θ_1, Θ_2):
sigmoid_probs = sigmoid(x, Θ_1, Θ_2)
return np.array([[np.sum((y - sigmoid_probs) * x),
np.sum((y - sigmoid_probs) * 1)]])

def hessian(x, y, Θ_1, Θ_2):
sigmoid_probs = sigmoid(x, Θ_1, Θ_2)
d1 = np.sum((sigmoid_probs * (1 - sigmoid_probs)) * x * x)
d2 = np.sum((sigmoid_probs * (1 - sigmoid_probs)) * x * 1)
d3 = np.sum((sigmoid_probs * (1 - sigmoid_probs)) * 1 * 1)
H = np.array([[d1, d2],[d2, d3]])
return H``````

``````def newtons_method(x, y):
"""
:param x (np.array(float)): Vector of Boston House Values in dollars
:param y (np.array(boolean)): Vector of Bools indicting if house has > 2 bedrooms:
:returns: np.array of logreg's parameters after convergence, [Θ_1, Θ_2]
"""

# Initialize log_likelihood & parameters
Θ_1 = 15.1
Θ_2 = -.4 # The intercept term
Δl = np.Infinity
l = log_likelihood(x, y, Θ_1, Θ_2)
# Convergence Conditions
δ = .0000000001
max_iterations = 15
i = 0
while abs(Δl) > δ and i < max_iterations:
i += 1
g = gradient(x, y, Θ_1, Θ_2)
hess = hessian(x, y, Θ_1, Θ_2)
H_inv = np.linalg.inv(hess)
# @ is syntactic sugar for np.dot(H_inv, g.T)¹
Δ = H_inv @ g.T
ΔΘ_1 = Δ[0][0]
ΔΘ_2 = Δ[1][0]

# Perform our update step
Θ_1 += ΔΘ_1
Θ_2 += ΔΘ_2

# Update the log-likelihood at each iteration
l_new = log_likelihood(x, y, Θ_1, Θ_2)
Δl = l - l_new
l = l_new
return np.array([Θ_1, Θ_2])   ``````