Right answer is (a) Characteristic equation
To explain I would say: A non zero solution will occurs when (A-ɅI) is a singular matrix or det(A-ɅI)=0 it is a characteristic equation. A characteristic equation is the equation which is solved to find the Eigen values, also called the characteristic polynomials.