The total extragalactic background radiation can be an important test of the global star formation history (SFH). Using direct observational estimates of the SFH, along with standard assumptions about the initial mass function (IMF), we calculate the total extragalactic background radiation and the observed stellar density today. We show that plausible SFHs allow a significant range in each quantity, but that their ratio is very tightly constrained. Current estimates of the stellar mass and extragalactic background are difficult to reconcile, as long as the IMF is fixed to the Salpeter slope above 1 M⊙. The joint confidence interval of these two quantities only agrees with that determined from the allowed range of SFH fits at the 3σ level, and for our best-fitting values the discrepancy is about a factor of 2. Alternative energy sources that contribute to the background, such as active galactic nuclei (AGN), Population III stars or decaying particles, appear unlikely to resolve the discrepancy. However, changes to the IMF allow plausible solutions to the background problem. The simplest is an average IMF with an increased contribution from stars around 1.5–4 M⊙. A ‘paunchy’ IMF of this sort could emerge as a global average if low-mass star formation is suppressed in galaxies experiencing rapid starbursts. Such an IMF is consistent with observations of star-forming regions, and would help to reconcile the fossil record of star formation with the directly observed SFH.