Solution

This problem deals with the so-called Compton effect: a photon scatters off a free electron, transfers part of its energy to the electron, and its own frequency decreases. This topic is very popular with various olympiad authors. For example, a similar problem appeared in the Belarusian selection for IPhO.

Let us immediately note: the masslessness of the photon and the constancy of the electron's rest mass have not been cancelled. Savchenko uses mass as a measure of the $\textbf{total energy}$ of these particles. Thus, $m = \frac{h\nu}{c^2}$, where $\nu$ is the photon frequency. Or, if using mass in eV, $m = h\nu = E_{photon}$, $m_e = E_{electron}$ — this can significantly simplify the notation.

We will use this notation, although some formulas will look a little unusual.

Then the conservation of energy:
$$m_e+m=m'_e+m_{\gamma}\tag{1}$$
Immediately we obtain the "mass" of the electron after the collision:
$$m'_e=m_e+m-m_{\gamma}\tag{2}$$

Relativistic invariant for the electron:
$$m'^{2}_e=p^2_e+m^{2}_e\tag{3}$$

Let the $Ox$ axis be directed along the initial direction of the photon, and $Oy$ perpendicular. Then the angle between the direction of motion of the electron after the collision and $Ox$ is $\varphi$. Conservation of momentum along the axes:
$$Ox: \quad \frac{m}{c}=p_e\cos\varphi+\frac{m_{\gamma}}{c}\cos\alpha\tag{4}$$
$$Oy: \quad 0=p_e\sin\varphi+\frac{m_{\gamma}}{c}\sin\alpha\tag{5}$$
We are not very interested in where the electron goes (the angle at which the photon is observed is practically important), so we eliminate $\varphi$. To do this, first substitute (3) into (4) and (5), multiply by $c$ and square:
$$(m-m_{\gamma}\cos\alpha)^2=(m'^{2}_e-m^{2}_e)cos^2\varphi\tag{6}$$
$$m_{\gamma}^2\sin^2\alpha=(m'^{2}_e-m^{2}_e)sin^2\varphi\tag{7}$$
Using the Pythagorean identity:
$$(m-m_{\gamma}\cos\alpha)^2=(m'^{2}_e-m^{2}_e)-m_{\gamma}^2\sin^2\alpha \tag{8}$$
Simplifying,
$$m^2-2mm_{\gamma}\cos\alpha+m_{\gamma}^2=m'^{2}_e-m^{2}_e \tag{9}$$
$(2)\to(9)$
$$mm_{\gamma}\cos\alpha=m_em_{\gamma}+mm_{\gamma}-mm_e \tag{10}$$

$$m_{\gamma}=\frac{m}{1+\frac{m}{m_e}(1-\cos\alpha)}<m\tag{11}$$

Note: If we return to the usual energy instead of mass, we obtain the decrease in frequency mentioned at the beginning and get the standard formula for the Compton effect:
$$\frac{1}{\nu'}-\frac{1}{\nu}=\frac{h}{m_e c^2}(1-\cos\alpha)$$
or
$$\Delta\lambda=\lambda'-\lambda=\frac{h}{m_e c}(1-\cos\alpha)$$