This is what a recent study set out to discover: the link between physical activity and how long it took for women to get pregnant. The research, which just appeared in the journal Fertility and Sterility, studied 3,600 Danish women between 18 and 40 who were planning to get pregnant and weren't getting any kind of fertility treatment. They were surveyed over a year about how many hours per week they exercised, whether their workouts were moderate or vigorous, and whether they got pregnant and if so how long it took them to conceive.
For many women, frequent, vigorous physical activity was associated with taking a longer time to get pregnant. In women under 30 who exercised vigorously less than once a week, the researchers noted 353 pregnancies over the span of the study. For women in the same age group who did vigorous exercise five or more times a week, there were just 85 pregnancies. The fertility rate "was lowest for the women who engaged in 5 or more hours [weekly] of both moderate and vigorous exercise."
The exception was women who were overweight or obese; for these women, any type of physical activity slightly helped, or at least did not hurt, fertility.
The study also found a modest increase in fertility regardless of a woman's weight if she did moderate exercise, and concluded that slim women who switch from vigorous exercise to a more moderate intensity while they're TTC may have more luck getting pregnant sooner.