These two passages seem to touch on this subject as well:
PD26: The desires that do not bring pain when they go unfulfilled are not necessary; indeed they are easy to reject if they are hard to achieve or if they seem to produce harm.
VS21: Nature must be persuaded, not forced. And we will persuade nature by fulfilling the necessary desires, and the natural desires too if they cause no harm, but sharply rejecting the harmful desires.
I find it interesting that, at least in St-Andre's translation, harm is referred to generally and not as specific to the individual. This, to me, correlates to what Don has described above. Basically, to achieve maximal pleasure you must consider the consequences of your actions.