Yeah, the lack of people walking amongst us from a future date, with future technologies and cool futuristic hairstyles might mean it will never happen. Which could either mean it is not possible or that we never made it far enough as a species to discover it.
Or you can believe that time travel only works by creating parallel worlds. Whenever anyone goes back in time a new branch forms so that the time line they are leaving is preserved. You then have to accept that we are in either the original timeline (un-likely), that the timeline we were in was changed in a way that we can percieve (for example someone went back in time, creating a new timeline, and then was immediately eaten by a dinosaur, causing no lasting effects that we could notice), or that time travel isn't possible. http://en.wikipedia.org/wiki/Many-worlds_interpretation#Time...
That's one of the things Steven Hawking has been known to say: the lack of time travelers among us suggests there is no way to time travel, at least not into the past. Apparently, what he said was "I gave a party for time-travellers, but I didn't send out the invitations until after the party. I sat there a long time, but no one came."
Maybe time travelers don't think we are interesting.
I think the problem with time travel is that is you went back in to to kill say Hitler.
Killing him would negate your reason for going back in time and cancel your trip. Accidental changes in history might not cancel trips but also might not be noticed for the same reason. History might at least seem immutable and hence the Travelers only have reason to observe not interact.