This study evaluated the relationship between pregnancy rate and semen cadmium concentration. This prospective and nonrandomized clinical study analyzed 341 male partners of infertile couples undergoing infertility evaluation and management. Semen samples were collected to analyze semen quality and cadmium concentrations. The main outcome was pregnancy during 60-day infertility treatment. Simple linear regression analysis revealed an association between semen cadmium concentration NS sperm count (r = −0.150, P = 0.0416) in nonsmoking subjects (n = 184). In both smokers and nonsmokers, semen cadmium concentrations were significantly higher in non-pregnant patients than in pregnant patients. In nonsmokers, Cox multi-variable fertility ratio analysis demonstrated an association between semen cadmium concentration and fertility (fertility ratio of log semen cadmium = 0.24; 95% confidence intervals (CI) = 0.12–0.47, P < 0.0001) after adjusting for related variables. Each tenfold increase in semen cadmium concentration was associated with a 4.17-fold increase in infertility ratio in nonsmoking patients. In smokers, Cox multi-variable fertility ratio analysis demonstrated that sperm count and semen cadmium concentration are associated with fertility (fertility ratio of log semen cadmium = 0.17; 95% CI = 0.04–0.63, P = 0.0085) after adjusting for related variables. In smokers, each tenfold increase in semen cadmium concentration was associated with a 5.88-fold increase in infertility ratio. In conclusion, low levels of cadmium accumulation in semen may contribute to male infertility by reducing sperm quality.