Sexual desire and testosterone are widely assumed to be directly and positively linked to each other despite the lack of supporting empirical evidence. The literature that does exist is mixed, which may result from a conflation of solitary and dyadic desire, and the exclusion of contextual variables, like stress, known to be relevant. Here, we use the Steroid/Peptide Theory of Social Bonds as a framework for examining how testosterone, solitary and partnered desire, and stress are linked over time. To do so, we collected saliva samples (for testosterone and cortisol) and measured desire as well as other variables via questionnaires over nine monthly sessions in 78 women and 79 men. Linear mixed models showed that testosterone negatively predicted partnered desire in women but not men. Stress moderated associations between testosterone and solitary desire in both women and men, but differently: At lower levels of stress, higher average testosterone corresponded to higher average solitary desire for men, but lower solitary desire on average for women. Similarly, for partnered desire, higher perceived stress predicted lower desire for women, but higher desire for men. We conclude by discussing the ways that these results both counter presumptions about testosterone and desire but fit with the existing literature and theory, and highlight the empirical importance of stress and gender norms.