Solution

A floating object is stable from vertical disturbance. If pushed down, the object will go back up, because there is additional buoyancy, and vice versa. Thus, we consider only tilting disturbance that does not change buoyancy. In other words, a horizontal line through point $O$ in the object at the same level as the water surface remains stationary so that the volume of water replaced by the object does not change. Before the object is tilted, the weight of the water replaced by the object is the same as the weight of the object (and remains so after the object is tilted), and thus

$$\rho_0\,a^2\cdot OC\cdot g=\rho a^2bg\qquad\Rightarrow\qquad OC=\frac{\rho b}{\rho_0}.$$

Consequently, $BC=\rho b/(2\rho_0)$, where $B$ is the center of mass of the water replaced by the object before tilting. After the object is tilted to the left by a small angle $\theta$, the center of mass of the water replaced by the object is at point $D$. This is a result of the body of water in gray being shifted to the body of water in dark blue, both of which have mass

$$\rho_0\cdot\frac{1}{2}\cdot a\cdot\left(\frac{a}{2}\right)^2\tan\theta\approx\frac{\rho_0\,a^3\theta}{8}$$

and their centroids at the distance approximately $a/3$ from point $O$. Thus,

$$BD\approx\frac{2(\rho_0\,a^3\theta/8)(a/3)}{\rho_0\,a^2\cdot OC}=\frac{\rho_0\,a^2\theta}{12\rho b}$$

and is almost perpendicular to $BC$ (as $\theta$ is small). The object will be tilted back to the right and restored to its equilibrium if buoyancy $\vec F$ (shown in green) is on the left of the weight $\vec W$ (shown in red) of the object; that is, if

$$BD>AB\cdot\tan\theta\qquad\Rightarrow\qquad\frac{\rho_0\,a^2}{12\rho b}>AB,$$

where point $A$ is the center of mass of the object and $AC=b/2$. Since $AB=AC-BC=b(1-\rho/\rho_0)/2$, the condition for stability becomes

$$\frac{\rho_0\,a^2}{12\rho b}>\frac{b}{2}\left(1-\frac{\rho}{\rho_0}\right)\qquad\Rightarrow\qquad\frac{a}{b}>\sqrt{\frac{6\rho}{\rho_0}\left(1-\frac{\rho}{\rho_0}\right)}.$$