I think it is all in the "toil" over a joy that is only "now and again."
If I enjoy playing an instrument, and it is "low toil" and "frequent joy," then I am not his target.
I enjoy playing my banjo (looking into getting a bouzouki) -- but really only at the very end of the day when I have given my greater mental energy to greater things. I do not learn songs, only improvise, every second that I play is the song.