Sefer Kesav Sofer writes that this is the way of all those who start fights and engage in machlokes. They come with complaints about the Rov of the city and say, “In any other city, they would have gotten rid of him long ago. It is only because we are such nice and kind people that we let him be our Rov for this long and didn’t fire him a while ago because of his deficiencies.”
The assembly of Korach said the same thing. They came and said, “It is enough for you." It is enough that we allowed you to be our leader for this long. It is only because “the entire assemblage is holy” and kindhearted that we didn’t get rid of you until now. “Why do you raise yourself?” At the very least, you shouldn’t raise yourself even higher so that you won’t have such a big downfall when we do get rid of you.