Take a look at the dynamics on your team. Who is really in charge? Who's the most influential? I've encountered a lot of managers who take a backseat to team members ... team members who are aggressive, who are causing some sort of fear or intimidation to others on the team.
That's not to say that you can't step back and allow others their moments in the spotlight. In fact, leaders give other people opportunities to shine. No, what I'm talking about are the situations when managers are cowering. They're afraid. They don't think that they can maintain control of the situation when others are aggressive or dominant.
This comes from a lack of confidence. It comes from the misunderstanding about what it means to lead.
It's not about being popular. That is not the point of managing or leading. Being popular ultimately comes with being a people builder. Being popular as a manager or leader comes from being consistent and fair in setting expectations, and holding people accountable, and providing development opportunities so they can exceed those expectations and grow.
If you are in a position where people are looking for you to lead them, the only time to step back is when you're making room for others to soar.