Unfortunately any single term (other than "Epicurean" itself) is going to have its own set of difficult associations. Hopefully over time the situation will improve as people talk about Epicurean philosophy more in "normal" circles.
However I see a problem with that label because it has a double meaning (the belief that money and material possessions are the most important thing in life - which is not what Epicureans believe at all) and wondering if we could come up with a better label.
As for the "ghosts" issue, that strikes me as an example of how - despite the decrease in organized religion, at least in the USA - we still have a lot of people who refuse to give up on more generic "spiritual" ideas.