14.5.9. Two particles with masses $m_1$ and $m_2$, moving with velocities $v_1$ and $v_2$ directed towards each other at an angle $\alpha$, merge into one particle. Determine the mass $M$ and velocity $v$ of the resulting particle.
Solution
First, let us solve the problem in the framework of special relativity, since the condition does not specify that approximations are allowed, and the chapter is indeed called "Special Relativity"...
The gamma factor: $$\gamma(v) = \left(1 - \frac{v^2}{c^2}\right)^{-\frac{1}{2}}. \tag{1}$$
Energy conservation: $$\gamma_1 m_1 c^2 + \gamma_2 m_2 c^2 = \gamma M c^2,$$ or, simplifying slightly, $$\gamma_1 m_1 + \gamma_2 m_2 = \gamma M. \tag{2}$$
Write the momentum conservation along the axes (one axis is along $v_1$, the other perpendicular to it). Let the velocity of the final particle be directed at an angle $\varphi$ to the first axis: $$\gamma_1 m_1 v_1 + \gamma_2 m_2 v_2 \cos\alpha = \gamma M v \cos\varphi, \tag{3}$$ $$\gamma_2 m_2 v_2 \sin\alpha = \gamma M v \sin\varphi. \tag{4}$$
Writing the conservation laws in such problems is not difficult; the most important thing is to solve the resulting system carefully and quickly. One can do the following: square (3) and (4), add the resulting expressions, use the Pythagorean identity to eliminate $\varphi$, and substitute $\gamma M$ from (2): $$(\gamma_1 m_1 v_1 + \gamma_2 m_2 v_2 \cos\alpha)^2 + (\gamma_2 m_2 v_2 \sin\alpha)^2 = (\gamma_1 m_1 + \gamma_2 m_2)^2 v^2,$$ $$(\gamma_1 m_1 v_1)^2 + 2\gamma_1\gamma_2 m_1 m_2 v_1 v_2 \cos\alpha + (\gamma_2 m_2 v_2)^2 = (\gamma_1 m_1 + \gamma_2 m_2)^2 v^2,$$ $$v^2 = \frac{(\gamma_1 m_1 v_1)^2 + 2\gamma_1\gamma_2 m_1 m_2 v_1 v_2 \cos\alpha + (\gamma_2 m_2 v_2)^2}{(\gamma_1 m_1 + \gamma_2 m_2)^2}. \tag{5}$$
Strictly speaking, this is the answer. Substituting (5) into (2) gives the mass as well.
However, judging by the author's answer, it is assumed that for all velocities $\gamma \approx 1$. Then $$v = \frac{\sqrt{(m_1 v_1)^2 + (m_2 v_2)^2 + 2m_1 m_2 v_1 v_2 \cos\alpha}}{m_1 + m_2}, \tag{6}$$ $$M = m_1 + m_2. \tag{7}$$
If the author had stated this in the condition, one could obtain the same result much more simply: $$M = m_1 + m_2,$$ $$(m_1 + m_2)\vec v = m_1 \vec v_1 + m_2 \vec v_2.$$ Squaring and using the scalar product rules: $$(m_1 + m_2)^2 v^2 = m_1^2 v_1^2 + m_2^2 v_2^2 + 2m_1 m_2 v_1 v_2 \cos\alpha,$$ and then express the answer.