Human transmembrane proteins Serinc3 and Serinc5 are antiviral restriction factors that inhibit HIV-1 infectivity. In the absence of viral antagonism, Serinc3 and Serinc5 incorporate into the envelopes of nascent virions and inhibit the fusion of virions to the target cells. The HIV-1 virus counteracts the restriction of Serinc3 by downregulating it from the cell surface and thus excluding it from budding virions. This is orchestrated by the viral accessory protein Nef and involves hijacking of the clathrin adaptor protein complex 2 (AP2)-dependent endocytosis. The mechanistic details of Nef-mediated Serinc3 downregulation, however, have been enigmatic. In this work, we investigated and revealed the molecular determinants of Serinc3 modulation by Nef. Our results show that Nef recruits Serinc3 by binding to its N-terminal cytosolic tail. Furthermore, Nef residues important for Serinc3-binding in vitro, and for the exclusion of Serinc3 from virions, overlap with those required for Nef-mediated CD4 downregulation, suggesting great mechanistic similarities between the two functions of Nef. In addition to shedding light to the mechanism of Serinc3 antagonism, our work also highlights the conserved substrate-binding pocket of Nef as a molecular hotspot for inhibitor development and antiretroviral drug discovery.