The nucleation of amyloid fibrils from monomeric protein, catalyzed by the surface of existing fibrils, is an important driver of many disorders such as Alzheimer\'s and Parkinson\'s diseases. The structural basis of this secondary nucleation process, however, is poorly understood. Here, we ask whether secondary nucleation sites are found predominantly at rare growth defects: defects in the fibril core structure generated during their original assembly. We first demonstrate using the specific inhibitor of secondary nucleation, Brichos, that secondary nucleation sites on Alzheimer\'s disease-associated fibrils composed of A{beta}40 and A{beta}42 peptides are rare compared to the number of protein molecules they contain. We then grow A{beta}40 fibrils under conditions designed to eliminate most growth defects while leaving the regular fibril morphology unchanged, and confirm the latter using cryo-electron microscopy. We measure both the ability of these annealed fibrils to promote secondary nucleation and the stoichiometry of their secondary nucleation sites, finding that both are greatly reduced as predicted. Re-analysis of published data for other proteins suggests that fibril growth defects that expose monomer planes or other structural units may also drive secondary nucleation generally, across most or all amyloids. These findings could unlock structure-based drug design of therapeutics that aim to halt amyloid disorders by inhibiting secondary nucleation sites.