You already build relationships with your team, but you find it hard to build a relationship with a customer. Perhaps someone has repeated the old saying "the customer is always right" once too often and you are now afraid to stand up for yourself: that's actually not uncommon, and customers know it too. That's why they sometimes act the way they do.
My view is that if you want to be a PM and the role needs you to deal with customers, there are strategies to cope with that. Assuming your company also employs sales people or people with "Account Manager" as part of their formal role, why not take them into customer meetings with you? It would be essential to brief them about the issues in advance, and most importantly, work out a strategy for dealing with the difficult situations that will inevitably arise, and predict how your customers are likely to react (especially to bad news). Think about how you want to respond to customer behaviours: maybe you can be flexible sometimes, while at other times, you need to put your foot down.
But also look for positive achievements that you can tell your customers, and win their trust. If you are only giving bad news, it's really tough out there. So don't just give bad news. Tell them about what you have achieved, what opportunities they will get from the things you are delivering, and make suggestions as to how to progress the project. Also ask for their help. People like to offer assistance, and it starts to make them part of your team, solving problems, instead of just throwing rocks at you while you sit and absorb the criticism.
So, back to your question: does an agile PM need account manager skills? - Yes, inevitably, but you can develop coping strategies and subcontract these skills to someone else until you have built your own confidence and can create the good working relationships that actually make the role feel rewarding. Use your wider team, and use the same interpersonal skills that make you a good delivery manager, and it will get easier.