Solution

If the cavity has radius $r$, then the center of mass (C.M.) moves on a circle of radius $r/\sqrt(2)$. Since it is initially at rest, the total force (shown in green) must be tangential to this circle. Consequently,

\begin{equation} N=2mg-N', \end{equation}

and the total force has magnitude $\sqrt2N$. The C.M. has linear acceleration of $\sqrt2N/m$ and angular acceleration

$$\alpha=\frac{\sqrt2N/m}{r/\sqrt2}=\frac{2N}{mr}$$

around the center of the cavity. This angular acceleration matches that of the rotation of the dumbbell around the C.M., which can be obtained by dividing the total torque by the moment of inertia around the C.M.

$$\alpha=\frac{(N'-N)r/2}{2m(r/2)^2}=\frac{N'-N}{mr}.$$

Equating the two expressions for $\alpha$, we have $N'=3N$. Using this result with (1), we have $N=mg/2$ and $N'=3mg/2$.