A Fool Loses What He Is Given
"It really is wondrous in my eyes," the Vayechi Yosef wrote, "how it is possible that people should be lax about answering amen, despite the words of Chazal extolling the value of this mitzvah. Regarding the laxity in amen one can use the words of Chazal (Chagigah 4a): "Who is a fool? The one who loses all that is given to him"!
The truth is that people are sometimes swept up with the masses who are lax in this mitzvah, especially as at times, we see respected people acting this way. But we need to read the words of Chazal (Yoma 84b) about this: "Regarding pikuach nefesh we do not follow the majority." With improper behavior, such as being lax about answering amen, which is a danger to the soul, one should not follow the majority, rather a person needs to take a stand and be strict to answer amen according to halachah (Vayechi Yosef Moadim 1, Shabbos Shuvah p. 198).
