וּתְשׁוּבָה וּתְפִלָּה וּצְדָקָה מַעֲבִירִין אֶת רֹעַ הַגִּזְרָה
In Parshas Ki Savo, the Torah concludes the kelallos with the reason why Hashem would punish us: תַּחַת אֲשֶׁר לֹא עָבַדְתָּ אֶת ה' אֱלֹקֶיךָ בְּשִׂמְחָה וּבְטוּב לֵבָב מֵרֹב כֹּל - For you did not serve Hashem with joy and a good heart, when you were blessed with plenty. This pasuk emphasizes that we must serve Hashem with joy. This means that all our mitzvos must be done with simchah including the mitzvah of teshuvah.
This may seem like an oxymoron to some, as part of the process of teshuvah is to mourn and regret our sins. Yet in truth there is really no contradiction, every mitzvah can be done with joy including the Torah commandment of teshuvah. The Baal Hatanya clarifies that while physiologically, regret and joy are conflicting emotions, in avodas Hashem it is no contradiction. He quotes the words from the Zohar: בְּכִיָּה תְּקִיעָא בְּלִבְּאֵי מִמִּטְרָא דָא, וְחֶדְוָה תְּקִיעָא בְּלִבְּאֵי מִמִּטְרָא דָא - a weeping etched in the heart from one side, and joy etched in the heart from the other side. (ליקוטי אמרים - תניא פרק לד)
Rabbi Shimon Sofer, in his sefer Shir Me'on, writes that he always wondered why the Vidui is recited with a pleasant tune (as is the custom in many communities) when it seems we should be crying and mourning the many sins we have committed. He explains that this is certainly the mesorah. Teshuvah must be done with joy, as the Rambam says it is a mitzvas asei, therefore it must be performed b'simchah. (שיר מעון - יום הכיפורים)
Even more than by other mitzvos, by teshuvah simchah plays an ever-important role.
The Rambam says מַה הוּא תְּשׁוּבָה What is teshuvah? הוּא שֶׁיַּעֲזֹב הַחוֹטֵא חֶטְאוֹ וְכוּ' שֶׁיָּעִיד עָלָיו יוֹדֵעַ תַּעֲלוּמוֹת שֶׁלֹא יָשׁוּב עַל חֵטְא זֶה לְעוֹלָם (רמבם הל' תשובה פ"ב ה"ב) - it is that the sinner should abandon his sin until Hashem will testify that he will not return to this sin again for ever.
How can one achieve such a level of teshuvah where Hashem will actually proclaim that he will never commit this sin again? When a person is ecstatic about the immense gift Hashem has given him and has emunah in the great transformative power of his teshuvah, he will use all his willpower to acquire it. When he fully believes that his teshuvah was accepted and that he has been completely cleansed, he wishes to maintain this newly acquired clean state. Due to his joy about the gift of teshuvah, he desperately wishes to retain the gift and not return to his sins. Thus, teshuvah with simchah can bring a person to this high level where Hashem will testify that he will no longer stumble in this area.
On a similar note, the Baal HaTanya explains that in a battle, a זריז is assured victory, even over the strongest opponent; through doing mitzvos with zerizus we can triumph the yetzer hora. The highest level of zerizus can be achieved through simchah. When a person is enveloped in joy about a mission it leads him to fulfil it with great levels of zerizus. And alacrity is the biggest marker of success. If we approach teshuvah with the eagerness and exuberance that leads to zerizus, we will surely be successful in our teshuvah. (ליקוטי אמרים - תניא פרק כו)