I think it makes sense to evaluate before starting a friendship or romantic relationship whether there will be net pleasure or pain, but doing this inside of relationships seems to make them feel shallow and transactional, at least for me.
This is a solid observation, and Epíkouros addresses those exact points. In a nutshell, the Garden observes all friendships starting with two individuals providing each other with mutual utility. In the beginning, you're totally right. Friendship can be described as being "transactional".
But I think you already nailed the bigger picture in your thesis, which is that "doing this inside of relationships" lowers the value of the relationship. It really can, and its not acceptable, and Epíkouros provides assurance that we cannot betray our friends out of convenience. I think I can provide a few, practical example of this when it comes to sex, death, and betrayal.
Each friendship [is worthwhile], by virtue of itself, but is taken at first for the sake of advantage. (VS 23)
I do like the word "transactional" in the beginning. Regardless, though, a huge goal of friendship is beyond the satisfaction of temporary needs is the cultivation of long-term, loving relationships. To obtain the latter, we have to begin with the former, starting with "I scratch your back...". There's a huge difference between helping each other have an orgasm, versus making a family. Both of those things are happening, but the long-lasting, worthwhile motivation is the latter. (I don't necessarily mean having kids ... that's not for everyone and statistics are divided about that choice. I'll leave that to each person to decide if they're ever ready for that experience).
[F]riendship [arises] according to the needs; Nevertheless it will always be beneficial to offer friendship just as [as it is] for us to plant seeds in the earth, thus [friendship] itself cultivates those communities that [work together to] perfect the pleasures. (10.120)
The otherwise transactional nature of relationships develops into something more valuable. In this regard, practicing vulnerability and forgiveness and holding dialogue to resolve issues is key.
But, you're right, we should do our best to practice choice and avoidance with new personalities. It would be absurd to invite someone hostile into a safe space, and wreck your peace.
Neither should we accept the overeager for friendship, nor the reluctant... (VS28)
Hopefully, we can maintain friendly discourse with as many people as possible who are neither hostile to our interests, nor eager to take advantage of our kindness. Still, we are encouraged to distinguish one from the other, which is tricky, and requires trial-and-error:
The one who exudes no confidence from the outside is best to bring together a fellowship of friends and consequently befriends the most possible members of one’s own kind; and when impossible at least do not vilify as not of one’s own species; and so long as that was not possible, avoiding contact with them was possible, expelling them from thought so long as the latter of these practices proves to be useful. (Key Doctrine 39)
Though, to speak to your initial concern, I think that we have to practice vulnerability for the greatest gift of friendship, even if vulnerability means that we occasionally get our hearts broken, or get betrayed. The immortal good of friendship is worth taking a reasonable risk.
...but still one must take risks for the sake of friendship. (VS28)
Indeed, the wise person will "never abandon a friend" (10.120). The end result of cultivating friendship is so meaningful, so blessed, that it justifies taking personal risks for its sake. Since our love of friends survives death, trying to make friends is always worthwhile.
...in defense of loved ones will at some time be ready to die (10.120)
This is the big one that reassures me that the Epicurean approach to ethics disqualifies gross selfishness and manipulation for the sake of strictly personal gain:
We enjoy no pleasure in being dead. There is no future satisfaction in a dead-state that can motivate an Epicurean to die. A person might be tempted to lie, cheat, or steal for an object, money, or power, but there can no, personal motivation for being dead, because nothing can be acquired that would motivate someone to act. There can only be motivation for dying if it directly saves a loved one. Even if dying is awful, and even if the only pleasure received is just a fleeting hope that they'll survive, it's still worth it. The pleasure for which we strive in that moment is the shared pleasure of another who will carry the torch of your friendship after you are gone.
Granted, there's simultaneously personal avoidance of being emotionally wrecked. The person who betrays a friend condemns themself to permanent misery. Dying prevents this.
But it also prevents everything else (besides the survivors), so unless death means "paradise", we're not exchanging mutual utility with a friend. What we are doing, in those last moments, is giving them everything we've got, without the expectation that we will ever receive anything ever again. There's nothing lesser about this, and Epíkouros firmly approves of this choice.
Just to quickly address two of your other comments:
... they only have value in so that they’re instrumental to my pleasure,
Not with regards to choosing to die for a friend, but that's also a rare situation, so I don't mean to over-use that example. Consider this: you are so loved by another person, that you have become instrumental to their own complete happiness. In that regard, we gain pleasure from knowing that we give pleasure to others ("...better to give than receive", so they say). While it's still self-motivated, I am comfortable describing it as a higher purpose.
in doing this, I lower my experience of pleasure within these relationships.
I wouldn't say "lower" at all, unless you're straight-up objectifying a friend.
Sex is a good example, here. When it's good, both people are providing each other with the same satisfaction. When it's really good, it's that first condition, plus deep love, or the immortal good of friendship, or emotional security from that gift. When it's bad, one person is gaining more satisfaction from the other. When it's really bad, both people are using each other and not even trying to cultivate what could become a beautiful friendship. When it's brutal, one person is explicitly harming the other for their fleeting pleasure.
We definitely lower our experience of pleasure if we treat people like objects. But so long as we see them as fellow subjects, then our mutual transaction becomes something that is infinitely (literally, infinitely, because it survives the death of your friend) more valuable.
This is important, though, because people do change, and sometimes for the worst. No matter how much you love your partner, and no matter how much you've each sacrificed for the other, and no matter how many hours you've spent in couples therapy, any relationship that features domestic abuse has to stop. The health of the subject is more important than their relationship with someone who has become hostile, or has become violent.
Granted, if it's good in the first place, I doubt it will devolve into domestic abuse.
Still, anything's possible. Life is complicated, and families and relationships are everything, everything good, and everything bad. When things get bad, really bad, irrevocably bad, it's important to preserve one's own peace and leave a bad situation. Divorce is important. Without getting disrespectful or too graphic, we had a situation in our extended family where severe, very severe abuse happened from someone who needed to be jailed. Here, the Epicurean insistence of keeping yourself healthy is really, really important. (As it turns out, again, sorry to my family for even bringing it up, but it was religion that kept it going).
But if you've cultivated true love, it's a moot point. Sometimes you gain pleasure from not obtaining things for yourself, but giving them to others. Maybe someday, we'll die to give someone else a future, knowing that we are surrendering every happy thing that we could ever hope to experience again. A Tale of Two Cities comes to mind.
