The Doctor isn't only quiet because of his heartaches of no Tardis and being dumbed down to being human but also then proceeds to stare at Jenny about being taken into the Labyrinth. He knows Vaeros is in the center and that there's a reason to go in but it's also something The Doctor had been in before playing Hide and Seek with The Toymaker.

"I came from a Labyrinth...."