The definitive literature is The Scrum Guide.
Product Backlog refinement is the act of adding detail, estimates, and order to items in the Product Backlog. This is an ongoing process in which the Product Owner and the Development Team collaborate on the details of Product Backlog items. During Product Backlog refinement, items are reviewed and revised. The Scrum Team decides how and when refinement is done. Refinement usually consumes no more than 10% of the capacity of the Development Team. However, Product Backlog items can be updated at any time by the Product Owner or at the Product Owner’s discretion.
Product Backlog items that will occupy the Development Team for the upcoming Sprint are refined so that any one item can reasonably be “Done” within the Sprint time-box.
As the Scrum Master I would have several questions. What is occurring during Product Backlog refinement? What about the refinement is unproductive? What suggestions do the Development Team members have to address the frustration? What issues arise during the Sprint that may be addressed via improved refinement? What benefits or risks are possible if the entire Development Team does not participate? How are the Product Backlog Items represented? Is the Development Team working collaboratively or does each member take a Product Backlog Item and work in isolation?
The distributed team is and will continue to be an impediment. Technology can assist with collaboration, but there is nothing like sitting in close proximity to the rest of the team and conversing throughout the day.