The RT instability occurs when a dense liquid is laid on top of a tenuous fluid. The more exact mathematical condition is that the density gradient is opposite to the gravitational force.
In the next sections we will develop the growth rate in the linear stage.
Equation of continuity
Conservation of momentum
We can simplify the equation by replacing the velocity by a flow potential . Substituting in the equations of motion yields
We assume an initial configuration where a heavy fluid of density is on top of a lighter fluid of density (which occupies the region .
Next, we introduce a perturbed flow potential
From the continuity of the normal velocity on the interface between the two fluids . The Bernouli invariant should also be continuous across the interface
where the square brackets denote difference across the interface. The pressure is continuous across the interface, so it vanishes. The kinetic energy term also vanish because it is of a higher order in the perturbation. The position of the interface is
We are thus left with a dispersion equation
Since the frequency is complex, the amplitude of this mode grows exponentially if a heavy fluid is on top, and decays if the lighter fluid is on top.
There is a more intuitive way to understand the this growth rate. Let us focus on a single cell of size near the interface between the two fluids. Let us denote by the position of the interface. The entire cell can be thought of as moving as a single body whose mass is , where $L$ is in dimension along the z axis. The force driving this cells is given by Archimedes' buoyancy force , so the equation of motion for the cell is
and the growth rate is