Wireframes and Prototypes
It seems to me that the presenting issue here is that you haven't clearly identified (or perhaps effectively communicated) the purpose of your wireframe phase. In my personal experience, wireframes are most useful for asking:
Is this user-visible feature set sort of what you had in mind?
If the answer is yes, then you have a starting point for discussions about cost and complexity. If the answer is no, then you need to revisit your specifications or assumptions.
Possible Phase Transitions from Wireframing
If you have a clearly-defined wireframe phase, you shouldn't be jumbling it in with your development. The output of each wireframe phase should be input to another milestone. For example, a wireframe phase may lead to:
- Another wireframe iteration.
- A return to requirements-gathering.
- The start of a level-of-effort estimation phase.
- Directly to a development iteration.
All of these are valid things to do, but I'd personally try to avoid going directly from a wireframe iteration to a development iteration without a distinct estimation phase that takes the wireframe as input.
Estimation and Communication Are Essential
Only the development team can provide accurate estimates. The goal is to use a client-approved wireframe as input to that estimation process. That allows you to have an honest discussion with the client about the trade-offs involved in design decisions, and to give them sliders to control time, quality, and scope.
Here are your main concerns, along with how estimation and communication can help.
Features are in conflict with specifications with other features.
Conflicts in the specification should be clearly communicated. You don't have to "prove" them; you're offering a professional opinion, and hopefully giving the client the benefit of your expert advice in how to avoid the conflict.
Features will result in poor user experience.
Again, a client is usually paying for your expertise. Requirements-gathering and wireframing aren't meant to be one-way conversations; they are discussions to iron out specifications and expectations. If you think a given feature will have a negative impact, discuss it with the client and document the assumptions and results of the conversation---perhaps even add it to your project's risk log if you have one.
Features are half-baked ideas that I know will not work if explored further.
This is part of estimation and planning. Again, communication is key. If an idea will be hard or expensive to implement, your post-wireframe estimations should reflect that. If an idea is "half-baked" (in the sense of being incomplete) then it needs to be sent back through the process for refinement. If you simply disagree with the decision, then provide constructive feedback and let the client make an informed business decision.
Poor Change Control is the X/Y Problem Here
[C]lients will complain how "such and such" is not user friendly, and throw a tantrum until we volunteer our time to upgrade their product.
This is your real issue. Everything else is trying to solve the problem without addressing the underlying cause.
If your process is transparent, iterative, and remains closely-engaged with your client throughout the project, then they will be able to make decisions and necessary corrections along the way. They are in the driver's seat; if they send you down a wrong path, it's their decision---but one they can easily correct in the next iteration.
However, if you treat specifications, wireframes, and other inputs as inviolate one-way communications, then you need to transfer all residual risk to the client. This is usually a contractual issue, and involves a lot of documentation and client sign-offs. This will usually prevent the need for unpaid re-work, but certainly won't build trust or rapport with a client.
Ultimately, the issue you're dealing with (e.g. how to avoid working for free) is a contractual change control issue. To fix the real problem, not just its symptoms, you need to inspect your contracting process, as well as how your company wants to handle change requests and how well it educates the client on the agreed-upon change control process.