I want to spent as little time as possible with them, they should just know what they have to do.
Stop and ponder this sentence. The other two answers have called attention to the attitude that is evident here. To me this conveys a fundamental disrespect for the team members in question and top down command driven management style. These indicate to me that I don't want to work on this team. In particular "they should just know" is a klaxon warning of huge project risk. The PM's job is 90% communications; if the PM wants to spend less time communicating, my spider sense is alerting me to a significant problem in the project. It is very unlikely that the Subject Matter Expert (SME) will ever "just know" unless the requirements are clearly communicated to them, and they are given the chance to explore the requirements.
In my personal opinion, the more planning I do with the SME, the greater the fidelity and assurance of my project schedule. Discussion with the SME tells me where the requirements need to be refined, where they are unclear, and where they are unrealistic. Discussion of the work package with the SME reveals my trade space when I have to make decisions about how to manage risk. My most common complaint is that SME are unwilling to talk about the requirements - I work to build up trust with them, so that they're willing to discuss, rather than just falling back into a defensive/conservative estimate.
If the SME are telling you that things don't work the way you've described, that is evidence of another problem. Granted, I will never understand the product as well as the SME's do, but when they tell me that I don't understand, that is a signal that it is time for me to listen - they are willing to share their expertise, and I have the opportunity to learn.
It is of course also possible that the SME's are simply irrationally resistant to change, and are expressing that resistance through a form of white mutiny. My response doesn't change - if they are irrationally resistant to change, then it is my role to get them to trust me - to convince them that changes will not be irrational or capricious. My role to ensure that their voice is heard in the change control process. That work package estimates will be based on their input, and that I will manage slack/risk to ensure that they can stay in a comfort zone between boredom and overwork.
Senior manager recently asked what I did while there was a SME in the room. I told him that "I keep the team's queue filled with interesting work." My goal is to keep delivering them interesting challenges that lead to satisfying rewards. Allocate the boring work so that everybody gets a fair share, and develop new work that enables them to stretch their skills without ever confronting the impossible. Enable them to deliver high quality work on time and on budget. (fortunately the SME smiled and understood). Point is not to glorify myself, but to outline that there are alternatives to the command structure implied in your question.