Sometimes the Doctor's smile falters a little and he gets this look in his eyes, and you can see how old he really is. It doesn't happen often, he doesn't like to show that face of himself, not to strangers, even less so to friends, especially not to him. But the Master's not stupid, really, he's just as smart as he is and knows him better than anyone else. The Master knows exactly what he's thinking about, when he gets that look into his eyes and stares off into the distance: that's the face he makes when he's thinking about someone or something that isn't here any more.
The worst thing is that the Master can't even ask what it is that the Doctor is longing for. He's too afraid that it will be something that the Doctor lost because of him, and that the Doctor will never forgive him that. Or worse, he's afraid that the Doctor will forgive him. Because, even if the Doctor does, he'll never quite be able to forgive himself. So when the Doctor's eyes grow tired and sad, the Master looks away and waits until the Doctor forgets again, even though he knows that the Doctor will never really forget what he's done, and all that he's destroyed, and the blood that's on his hands.
The Master will never be able to forget, either.