There is a lot of evidence that black holes follow the usual rules of quantum mechanics, and exhibit all of the fundamental properties of quantum systems such as a discrete set of energy levels. However, many of these properties are difficult to see directly in gravity; there is an apparent tension between the gravitational description of black holes in terms of smooth geometry and the discrete 'quantum' aspects of quantum mechanics. In this work we focus on understanding certain probes of the discreteness of the black hole energy spectrum which exhibit this tension, such as an averaged version of the two-point function of operators widely separated in time outside of a black hole in AdS space. The chaotic dynamics of black holes indicates that its energy levels should have random matrix statistics, and its energy eigenstates should obey the Eigenstate Thermalization Hypothesis. These expectations allow us to make precise predictions for the behavior of our probes. We find strong evidence that contributions from spacetime geometries with nontrivial topology, and even arbitrary numbers of disconnected components, are responsible for reproducing the expected behavior of our probes. Physically, these contributions correspond to including effects from the absorption and emission of closed 'baby' universes from the black hole spacetime. These processes have a small amplitude, but may provide the dominant contribution to the transitions between very distinct states of the black hole.