Translating Requirements with User Stories
Rather than attempt to answer the whole question, I'm going to answer just the part about translating requirements because I think it's often the most misunderstood. You asked:
Is there a good way to translate business requirements to software development requirements?
There are many ways to do this, but one of the most useful agile techniques is the user story. A user story contains perspective ("As a ..."), a deliverable ("I want ..."), and some context ("so that ...") that provides a solid framework for defining the scope and technical tasks that meet the business requirement.
Of course, as an agile technique, user stories are inherently iterative. Mike Cohn points out that:
[A] user story...is incomplete until the discussions about that story occur.
In other words, while user stories are a great starting point for delivering value, they don't replace the need for ongoing interaction and routine feedback between the development team and the end users (or their proxies, such as the Product Owner).
No matter how far down the rabbit hole you go, you still need a feedback loop. For example, converting user stories to acceptance tests with Cucumber often builds an initial agreement (ideally using domain-specific language appropriate to the business) on what the specification really means. However, it's still extremely common to uncover additional use cases or hidden requirements only after a requirement is met, which then requires refinement or refactoring.
Here's a concrete example. I had a client that specified that they wanted all records from a database table to be available on a single UI screen---no paging, no Ajax or remote procedure calls, just local access to all records delivered in a single batch to the client. The feature was delivered, having met all the acceptance criteria originally specified...and was then redesigned because it didn't meet the performance expectations of the end users.
Sometimes this happens because the end result is unexpected or unanticipated, and sometimes it happens because the organization accepts business risk, betting that a shortcut will pay off. Whatever the reason, though, it's common enough that continuous refinement and re-evaluation are baked into agile development frameworks.
Requirements Should Be Living Documents
Most of the problems with project management come from treating the requirements document (whether a formal specification or a set of user stories) as an historical artifact. User stories make treating requirements as living documents more intuitive, and encourage informed discussions about scope, risk, and usability.
If end users and developers don't remain routinely engaged with one another in a tight feedback loop, it pretty much ensures that:
- The business will ask for the wrong things.
- The development team will deliver unusable things.
Requirement documents are, from one point of view, a proxy for end-user expectations. If you build a good feedback loop into your project and revisit the requirements throughout your project life-cycle, you will be in a better position to manage those expectations successfully.