A ghost cannot haunt you if you don't believe in ghosts. You'd simply walk around your 'haunted' house like a champion, ignoring the knocks in the night, attributing them to wind. Who cares?
It's only the believers who suffer fear. Belief gives power.