Given,
\(\frac{p^{2}}{a^{2}}+k \cos \alpha+\frac{q^{2}}{b^{2}}=\sin ^{2} \alpha......\)(i)
\(\cos ^{-1}\left(\frac{p}{a}\right)+\cos ^{-1}\left(\frac{q}{b}\right)=\alpha\)
As we know,
\(\cos ^{-1} x+\cos ^{-1} y=\cos ^{-1}\left(x y-\sqrt{1-x^{2}} \cdot \sqrt{1-y^{2}}\right)\)
\(\cos ^{-1}\left(\frac{p q}{a b}-\sqrt{1-\frac{p^{2}}{a^{2}}} \sqrt{1-\frac{q^{2}}{b^{2}}}\right)=\alpha\)
\(\cos \alpha=\left(\frac{p q}{a b}-\sqrt{1-\frac{p^{2}}{a^{2}}} \sqrt{1-\frac{q^{2}}{b^{2}}}\right)\)
\(\frac{p q}{a b}-\cos \alpha=\sqrt{1-\frac{p^{2}}{a^{2}}} \sqrt{1-\frac{q^{2}}{b^{2}}}\)
Squaring both sides, we get
\(( \frac{p q}{a b}-\cos \alpha)^2=(\sqrt{1-\frac{p^{2}}{a^{2}}} \sqrt{1-\frac{q^{2}}{b^{2}}} )^2\)
\(\frac{(p q)^{2}}{(a b)^{2}}+\cos ^{2} \alpha-2 \frac{p q}{a b} \cos \alpha=\left(1-\frac{p^{2}}{a^{2}}\right)\left(1-\frac{q^{2}}{b^{2}}\right)\)
\(\frac{(p q)^{2}}{(a b)^{2}}+\cos ^{2} \alpha-2 \frac{p q}{a b} \cos \alpha=1-\frac{p^{2}}{a^{2}}-\frac{q^{2}}{b^{2}}+\frac{(p q)^{2}}{(a b)^{2}}\)
\(\sin ^{2} \alpha=\frac{p^{2}}{a^{2}}+\frac{q^{2}}{b^{2}}-2 \frac{p q}{a b} \cos \alpha.....\)(ii)
Comparing equation (i) and (ii), we get
\(\mathrm{k}=-\frac{2 p q}{a b}\)