In general, I agree with Chef. I have some of those exes in my past too. The ones I just can't help but say, "What if...?" It's very normal, impossible to avoid, and fairly innocuous.
That said, this still concerns me a bit. Daydreaming every once in a while about "what-ifs" and e-mailing them...