Hmmm.
I visited him a second time at Granite Falls, but still didn't get the Ask to move in option, so I went home.
When I opened my relationship panel (while at home) and clicked "Friends," the hermit showed up. So I clicked on his picture and selected "Travel."
This is different from starting out by selecting "Travel with..." Sorry I explained incorrectly the first time--I don't have my game open right now.
Also, if I remember correctly, he doesn't show up all the time (that is, I didn't catch him at home at first).
I remember being irritated and wondering why he wouldn't be available since he didn't have a job, lol. I guess he likes to hang out.