First, some clarification is important. Scrum does not expressly forbid any job. People in the Scrum Team can have any job titles as long as it respects the structure and rules of Scrum. Further, people can exist outside of the Scrum team that support the team as long as it does not violate the rules in Scrum. Now, I've worked in Scrum teams who are supporting with a Project Manager or who have very engaged managers while still allowing for self-organization. So, the question really becomes, is it better for the team to be self-organizing or directed?
As you point out, Scrum definitely advocates for and demands self-organization over direction.
The goal of Scrum is not to optimize cost. In fact, it uses empirical process control, which is by far the least efficient form of process control. However, it is the most highly effective form of process control for solving new problems. It was conceived in a time where a vast majority of knowledge work where projects were solving new problems were using an efficient, defined process and those processes were failing for years on end. So, first thing to understand is that the purpose of Scrum is optimized to solve problems quickly.
To that end, it suggests a small self-organizing team. Some key reasons include:
Communication channels: The number of communication channels can be found with this formula: c = n*(n-1)/2
. This means around 10 people becomes a major tipping point. Loads of other research supports this and actually supports even smaller teams of around 5 - 7.
Empathy: When you are trying to solve someone's problems, you need a lot of empathy for the person you're solving the problem for. Self-organization promotes ownership and a connection to the customer. Directive management promotes compliance to a task. Therefor Scrum favors self-organization.
Process iteration: Scrum does not just iterate on the product, it iterates on how the process is done. This can just be done way faster by a self-organizing team.
Creativity: Directive management focuses on extrinsic motivation. Self-organization has a mix of extrinsic and intrinsic motivation. Research has shown that extrinsic motivators constrain thinking and inhibit creative solutioning. Again, given what scrum optimizes for, that explains its lean toward self-organization.
These are a few reasons off the top of my head. I'm sure there are many more. I have to stress though that managers, team leads, and project managers do not have to be directive. I've seen plenty of people in those jobs support self-organizing Scrum teams and violate none of the Scrum rules.