Right choice is (b) 0

Explanation: A generalized Eigen value problem is represented by homogeneous matrix equation, [K- ω^2M]X=0. From matrix equations methods, the equation has a non-zero solution for X if the determinant of the matrix [K-ω^2M] equals to zero.