Seeing as your question asks 'ideally' I would say 'Product Development', along with the rest of the development team and product owner. Realistically, this isn't a department that exists in most organisations and would probably only work on a smaller scale.
This is very much dependent on the organisational structure, but I think a Scrum Master should sit under the 'Technology' or 'IT' umbrella, rather than HR. Depending on the size and structure of the company, this could be under a head of development (if that is all there is) or a Head of Process, if there is also a Head of Dev and Head of Test.
I'm not quite sure what 'product' represents in your organisation, but if people have a good understanding of agile and product development means making a good product, rather than cramming more features; it could be the right place.
Where do the developers sit - in 'Product'? A scrum master is an inherently technical role and developers (and their managers, one would presume) are the people you will be working with. A lot of the challenges I have faced as a Scrum Master would be fairly meaningless if I tried to discuss them with a manager in HR, while a technical manager (in my case the CTO, in my last role in a larger organisation Head of Development) has a fairly good understanding.
Interaction with HR is important from a people/relationship point of view, but you could argue the same with any 'Manager' role - it doesn't mean managers should belong in HR. As an aside, I am not saying a scrum master is a manager, but does face similar challenges around people, motivation etc.
Also, you say
as HR does not have any direct agendas when it comes to product work
A Scrum Master does have an agenda when it comes to product work - you want the team to be effective, not just 'happy'. As I am sure you know, productive doesn't just mean more features, so the agenda is different from that of a more sales oriented Product Manager, but still aligns with Product Development - this is very different from the objectives of HR.