It might help to separate backlog refinement and sprint planning. Backlog refinement might be attended only by people working on the product concerned.

If developers can work on multiple products, it might help to roughly define in advance how much (e.g. 30%) time/work should be spent for one product.

Talk about this problem in a Sprint Retrospective.

we've started tracking velocity as a whole team instead of as individuals, but I'm not sure if this will be an issue considering the team set-up

As long as it work I'd keep this if you want to be a team. Tracking the velocity of single developers might not help nor making them happy.

It might help to separate backlog refinement and sprint planning. Backlog refinement might be attended only by people working on the product concerned.

If developers can work on multiple products, it might help to roughly define in advance how much (e.g. 30%) time/work should be spent for one product.

Talk about this problem in a Sprint Retrospective.

we've started tracking velocity as a whole team instead of as individuals, but I'm not sure if this will be an issue considering the team set-up

As long as it works, I'd keep this if you want to be a team. Tracking the velocity of single developers might not help nor making them happy.

Edit: Typos and what I want to add: This situation is not good but I understand that sometimes there is no choice and one has to find a solution :-)