Hmm. Well, I do think that taking their number and telling them you'll call them when you're ready would be enough for many home teachers (or telling them you'd prefer no visits but would be ok if they wanted to put something in the mail). They're just trying to maintain contact so that if you feel you'd like to start going back to church then you have a link, a connection, an easier way to return.
One thing to think about, though, is that having home teachers that visit you isn't a commitment to return to church and might not be a bad thing. They're a resource for help if you need it and the visits should be brief (with a short spiritual message).
As far as finding you is concerned...I'm not sure exactly how records for inactive members get around and end up in the ward they would be attending. Inspiration?
Marianne