TL;DR
The entire team needs to be present during Sprint Planning. This includes the Product Owner. However, only the Development Team actually estimates the user stories.
Sprint Planning
Sprint Planning is where the bulk of Scrum estimation is performed. While the Product Owner doesn't actually participate in the estimation itself (e.g. he doesn't get to vote during Planning Poker), he is part of the estimation process so that he can answer questions about the stories being estimated, help the team refactor stories, or adjust the Product Backlog priorities in real-time during the planning session.
Who Should Estimate
The entire development team needs to participate in the estimation process, even on stories where they will be playing a secondary role. The reason is two-fold:
Everyone has a role to play in a well-written story.
A client-side story still needs data from the server side, and vice-versa. A programming story needs testing, documentation, and deployment. A story that doesn't involve most of the team is often a project smell that you have the wrong team in place, don't have a cross-functional team, or have a flawed user story.
A consensus estimate is usually more accurate.
By ensuring the whole team is involved in the estimate, everyone's skills can be brought to bear. Will your server-side programmers always know what their API changes will do to the client-side? How much technical writing will be involved in this new feature? Are there any required changes to the testing infrastructure that the QA guys need to implement to test this new feature?
Making sure that the whole team agrees to an estimate means that all eyes have been put on the task, and that the entire spectrum of team skills are in play. Tasks can still be mis-estimated, but you should find that you have fewer unforseen problems that could have been avoided by having the right people look at it during Spring Planning.
Remember, part of Sprint Planning is to decompose stories into tasks for the Sprint Backlog, and add additional tasks to ensure that the user stories meet the "definition of done." You are more likely to get this right when all the players are involved.
The Necessity of a Whole-Team Approach
The holistic approach to estimating and delivering features is the cornerstone of agile techniques. Whole-team estimation keeps the project planning process transparent and visible to everyone, increases intra-team communication and buy-in, and is essential to the Scrum process. If you aren't taking a holistic approach, I'd argue that the essence of the methodology has been lost.