Assignment 2
DSC 212: Probability and Statistics for Data Science
Due date: 5:00 pm PST, 13th March, 2023
1. Consider Bernoulli(p) observations 0 0 1 0 1 0 1 1. Plot the posterior distribution for p for the following
prior distributions Beta(1,1), Beta(10,10), Beta(1, 10). Mark the inflection points in the plots.
Note: Hand-drawn plots are sufficient.
2. Let X1, X2, . . . , Xn
i.i.d.～ Uniform(0,θ). Calculate the posterior distribution corresponding to the prior
density f(θ) ∝ 1θ .
3. In this problem, we derive a general approach—called iterative reweighed least-squares—for obtaining
the empirical risk minimization solution with a smooth loss, and unconstrained linear function class.
Consider minimizing a function L : Rd → R, and assume that L is continuously differentiable up to
second order. The Newton iteration for obtaining the minimizer of L is
θt+1 = θt [2L(θt)]1L(θt) (1)
where 2L and L are the Hessian and gradient of L respectively.
(a) Show that iteration (1) is equivalent to minimizing the second-order Taylor expansion of L around
θt.
i (2)
for weights αi(θ) and wi(θ) that you specify.
(c) Let X ∈ Rn×d be the matrix whose ith row is x?i . Let α(θ) = (α1(θ), . . . , αn(θ)) and Wθ =
diag(w1(θ), . . . , wn(θ)). Show that iteration (1) is equivalent to solving
θt+1 = argmin
θ∈Rd
∥∥∥W 1/2(θt)(Xθ ? z(θt)∥∥∥2 (3)
for z(θt) ∈ Rn which you specify. Thus Newton iteration in this context is equivalent to solving a
reweighed least-squares problem at each iteration.
Hint: L(θ) = Xα(θ) and 2L(θ) = X?W (θ)X.
(d) Consider the case where (t, y) = yt + log(1 + et) is the logistic loss. Show that in this case,
wi(θ) = σ(θxi)(1 σ(θxi)) and find an expression for zi(θ).
4. (Bayesian linear model) For the fixed design linear model with X = [x1, x2, . . . , xn]? with observations
yi = x=
i β + σεi having i.i.d. standard normal noise εi. Consider the likelihood and prior given below.
fY |β(y|w) ∝ exp(12σ2∥y Xw∥2) (4)
fβ(w) ∝ exp(∥w∥2Γ) (5)
Here Γ is a positive definite precision matrix and ∥w∥2Γ := w?Γw. Find the posterior distribution Pβ|Y .
5. If X ～ N (μ,Σ) where μ ∈ Rd and Σd×d is a positive definite matrix. Find the distribution of the vector
AX ∈ Rp for a fixed matrix A ∈ Rp×d.