But there are schools and people who say pleasure can lead you astray from a happy, fulfilling life, therefore it must be repressed, avoided, or rejected outright. Establishing it as the goal - that to which all else points - short circuits that argument.
Here is where I think you have to go back to the physics; you can't just jump into an ethical argument with people who don't accept your premises.
What ELSE besides pleasure do these people propose? (rhetorical question) Does that thing even exist in nature, apart from causing pleasure and pain for humans?