^{[The Doctor replies in a reassuring and calming voice]}

His name is Vaeros. He teleported all of us here. Into this garden. All of us.

More than likely; no time is passing in your universe and you're suspended here in this bubble universe. Why you're here is anyone's guess. Maybe Vaeros is lonely, despite having locals, already. Or it's just some sort of sick game.