When about to take on a project which is mostly staffed by volunteers, for a non-for-profit organization, what kinds of challenges should I prepare for, which I am unlikely to have seen elsewhere? Are there some books, courses, or other resources which will help me prepare for the transition?
The biggest difference between voluntary projects and typical ones is no power, which leaders can use over the team. It affects the way the project goes on many levels but the main differences would be:
There are no managers so people are asked to do something or declare they'd do something but can't be told to do that. It means the worst tasks are often understaffed or done poorly.
You have who you have - you can't just ask for a person with specific skills as you need them in the project. You have to fill the gaps with the group you work with. It's very important to have the group well-balanced.
Sometimes people just screw up and you can't do anything about that. People don't keep their word or just don't show up and you have no tools whatsoever to bring them back or make them do what they promise.
There's way more freedom in voluntary projects. Usually there's no client with big bucks who tells you what exactly they expect - there are many more decisions which are made by the team (or team members).
You can also see the post with my thoughts after running voluntary project (TEDx conference).
In terms of what need to prepare for:
Risk management. Have a backup plan for almost everything.
Look for people who you can count on. Few of them will do more than dozens of those who are quick to talk but not-so-quick to do.
Discussion facilitation. There will be plenty of discussion with different opinions thrown all over the table. But unless you make some decision you won't push the project forward, which sometimes mean you have to make a decision without taking into consideration everyone's opinion.
Roll up your sleeves. In voluntary projects everyone's a worker. There's rarely a leader who focuses purely on managing a project. Often leaders do most with their own hands.
Bone-up on your leadership skills.
More particularly, the ability to lead through inspiration, being well organized and the softer skill of talking people into things.
And patience. You'll need lots of patience.
The biggest challenge, of course, is unreliability of estimates. You can use "padding", which is the first and the wrong choice. Or you can duplicate resources, which is the right choice, but will make your life more difficult.