This is interesting because the most “realistic” (i.e. still not realistic) depictions of time travel in fiction involve travelling through a singularity or wormhole. So you probably have to be in space to start with, but also both ends of the wormhole have mass so they can be orbiting a planet or star and stay within a stable distance of it. It solves this particular problem (just leaving the other usual problem of causality!) It also proves your point since it does allow travelling in space, in fact it allows travelling faster than light.
I think the converse is true as well, that if faster than light travel is possible then time travel must be possible, at least if you take relativity at face value. As others have pointed out there’s no universal reference frame, and for any journey that is faster than light in one reference frame, there is another frame in which the journey goes backwards in time.
This is interesting because the most “realistic” (i.e. still not realistic) depictions of time travel in fiction involve travelling through a singularity or wormhole. So you probably have to be in space to start with, but also both ends of the wormhole have mass so they can be orbiting a planet or star and stay within a stable distance of it. It solves this particular problem (just leaving the other usual problem of causality!) It also proves your point since it does allow travelling in space, in fact it allows travelling faster than light.
I think the converse is true as well, that if faster than light travel is possible then time travel must be possible, at least if you take relativity at face value. As others have pointed out there’s no universal reference frame, and for any journey that is faster than light in one reference frame, there is another frame in which the journey goes backwards in time.