Here we explore how imperial structures contributed to set the level of inequality in two ancient empires, the Roman Empire ca. 165 CE and the Chinese Han Empire ca. 2 CE. We estimate the overall levels of imperial inequality as the combination of inequality between and within regions. We find that the Han Empire was, overall, more unequal and extractive than the Roman Empire. Other empires, however, were even more extractive, as shown by a comparison with the Aztec Empire ca. 1492. We argue that higher inequality increased the potential for political instability and the collapse of empires.