On the Absence of High-redshift AGNs: Little Growth in the Supermassive Black Hole Population at High Redshifts