There is a chilling effect when he is here. Am I overreacting? Should I just try to build up the confidence of the team? or should I ban my boss from the retrospective?
In my experience, this is a classic case of missing the forest for the trees, and mistaking process problems for interpersonal ones. Let's enumerate some of the issues that underpin your question.
- You think there's a "chilling effect" that is negatively impacting the Scrum Retrospective event.
- You are seeking validation from outside the team, rather than asking the team (individually, collectively, privately, or otherwise).
- You yourself are afraid to communicate about a perceived issue within the retrospective, or directly with the people involved.
- You have a "boss" on the team, rather than a Scrum Team member.
- Your boss is unclear on the difference between Scrum Team member (regardless of Scrum role) and line management responsibilities.
In short, you definitely have a communication problem within your process, but it all seems to boil down to a lack of trust and honesty. Fundamentally, if you can't be honest within the team or about reporting out process impediments, then you have a Scrum implementation failure. Scrum simply can't function effectively in a fear-based organizational culture, regardless of whether that culture comes from the team members or "tone at the top."
There is no silver bullet. Fear, uncertainty, and doubt will ruin any project management framework. This isn't a problem unique to Scrum. You either need to tackle the source of the team's fears head-on, accept things as they are, or move on as professionally as possible. To address the problem head-on, you and the rest of the team (including your boss) should keep the focus on the Scrum process rather than on interpersonal conflicts, whether real or perceived.
To kick-start the process improvement, you should take a hard look at yourself and your role as Scrum Master. If you perceive a problem, why don't you feel confident about collecting or facilitating honest feedback from the team? More importantly, if you're the Scrum Master (and therefore the person responsible for refereeing the Scrum process), why don't you feel safe in having an honest conversation with the "boss" (whatever his actual Scrum role may be) about his impact on the process?
If you can answer those questions honestly, you'll probably find your own answer. Based on past experience with dysfunctional Scrum implementations, the first step is almost always to improve communications. If you can't or won't do that yourself, then it's unrealistic to expect other team members (especially introverted, "junior," or inexperience team members) to do it either.
Fundamentally, if the whole team (including you) feels unsafe or insecure about being honest with one another, or communicating about process impediments with your organizational leadership, then it doesn't really matter whether your boss is the one at fault or not. The underlying issue is that the team collectively lacks cohesion, trust, situational awareness, and open communications. If the team can't collectively fix those things, then it can't fix the Scrum implementation either.
At the end of the day, you may not be able to change the social dynamics of the team. If that's the case, your project is likely to fail. If that happens, there will be plenty of blame to go around. On the other hand, if you decide to have an open and honest conversation as a team and your boss isn't willing to problem-solve along with everyone else, then he gets to keep both halves of a broken process.