I'm currently managing 7 teams that contribute to a single project. While we're one large "team", each team does their own estimates. This sounds very close to what you have -- one large team, made up of several smaller teams.
This is where your problem can be solved -- recognizing you have multiple teams. Each team should be responsible for their own ...
Never use tasks with time estimates. Ever. There are many reasons for this.
It's not Agile, for one. We use "working software" not "completed tasks" as a primary measure of progress
Estimating in absolute time is impossible. Even if it weren't, your time does not equal my time
It's a recipe for old command and control behavior
It is no different than a ...