What specifically is all the "stuff" a PM is actually responsible for?
A complete answer: "A Project Manager is responsible for Project Management". Project Management is a discipline (we can even call it a "science") with its body of knowledge explained in PMBOK (for example).
A short answer: a Project Manager is responsible for keeping a project in its scope, time, cost, and quality constraints.
Project management generally deals with planning and organizing a project and dealing with people and resources in a way which allows to reach project goals.
Specific tasks would depend on project's size and complexity and also on a specific method chosen as project management approach.
Planning is about preparing some kind of plan for the project. Depending on the project size and approach you use it can be anything from just a general list of project goals to a very detailed list of tasks planned to be done by specific people. Also you can spread planning over a longer period of time, so you just take whatever is the most important/most urgent coarse-grained task at the moment and you plan in a detailed way only work related to the task and you repeat the process over and over again.
Organizing is about getting specific people to do specific tasks. Again, much depends on a project. In agile projects often you base on self organization which means the team makes many decisions, like who does what, on themselves. In more formalized project it can be responsibility of PM, or PM along with team manager, to make those decisions. Also when I think about organizing a project I see reacting for changing situation, which includes risk management for example.
Dealing with people and resources means getting enough of them to deliver project within constraints. Most of the time the problem is in people, too few or too incompetent, but it's PM's job to react whenever something is wrong and project can't be completed because of insufficient or inappropriate people and/or resources.
Every PM's activity should be aimed to help with the general goal which is delivering the project on time, on scope and on budget. Basically if anything helps to do that it's the part of PM's job.