Poke him to tell you his schedule. And you two should figure out what can be done if you need alone-time. :)

(not sure about the next couple days, though, since there will be an extra person beyond Joe. You'll be ok?)