The Maharal translates the words, נבהלו כי מפניו , literally: "They were frightened by his face." Because if the pasuk means that they were afraid of him, it would say, ממנו נבהלו כי, "They were frightened from him."
The Maharal explains that they saw Yosef in a different light, as though he had a brand- new face that they never saw before, and this frightened them.
At first, the brothers viewed Yosef as a murderer, a rodef, someone who desires to harm them. This is the reason they believed they had permission to kill him or to sell him to Mitzrayim. But now they saw that they had erred. They saw a new Yosef, a new face. Yosef didn’t want to kill them. If he did, he could have done so several times since he was the ruler of Mitzrayim, and they were at his mercy. The brothers realized they were mistaken, and that's why they were so frightened. They were frightened by the magnitude of their error and how severely they had sinned.
There are times when we think someone is against us, someone is cruel to us, etc. but it is possible that we aren't seeing things correctly. Practice seeing this “new face” in the people around you, and then you will be friendly and at peace with everyone.
