Solution

We can get an estimate of the angular speed $\omega'$ of the outer cylinder as follows. In steady state, an average gas molecule must have tangential speeds of $\omega\,r_1$ and $\omega'\,r_2$ at the inner and the outer cylinders, respectively. We consider the motion of one such molecule whose velocity (shown in red) has magnitude $v$ and whose tangential velocity at each point is shown in pink. If the velocity vector makes an angle $\alpha$ (shown in blue) with the tangent to the inner cylinder, then $v=\omega\,r_1\sec\alpha$. Similarly, if the velocity vector makes an angle $\beta$ (shown in green) with the tangent to the outer cylinder, then $v=\omega'\,r_2\sec\beta$. Equating these two expressions for $v$, we have

$$\omega\,r_1\sec\alpha=\omega'\,r_2\sec\beta\qquad\Rightarrow\qquad\omega'=\omega\cdot\frac{r_1}{r_2}\cdot\frac{\cos\beta}{\cos\alpha}.$$

Using the law of sines in $\triangle OAB$, we have

$$\frac{r_1}{\sin(90^\circ-\beta)}=\frac{r_2}{\sin(90^\circ+\alpha)}\quad\Rightarrow\quad\frac{\cos\beta}{\cos\alpha}=\frac{r_1}{r_2},$$

and thus $\omega'=\omega\,(r_1/r_2)^2$.

Interestingly, if the gas is NOT rarefied and this problem is treated as a Taylor-Coulette flow, we still obtain the same answer.

Addendum: The assumption that an average gas molecule must have tangential speeds matching those of the inner and the outer cylinders means that there is no tangential momentum transfer. Therefore, conservation of angular momentum gives the desired result directly.