Something else that springs to mind is that ultimately this question comes down to hedonic calculus. If the “effort” you’re putting into the pursuit of pleasure is leading to more pain than pleasure, then one “should not” put that amount or type of effort into that particular venture.
I think that if you do a lot of "calculating" then there will be times that for a particular activity, the amount of pleasure produced breaks even with the amount of effort required...therefore you could reason with yourself that you might as well not do the activity.
Here is a practical example: There is a specific type of class that is being offered (it could be a dance class, a yoga class, or an art class, etc.) and you feel fairly certain that you will enjoy the experience and potentially meet new friends. However, the city that it is in will take an hour drive time (each way, so a total of 2 hours drive time). So even though the drive won't be particularly painful (just driving and maybe listening to an audiobook) ...the effort requires too much time. There actually isn't anything physically painful about the idea "it takes too much time"...it is just a mental judgment that it doesn't seem worth the effort.