Observe at Its Designated Times

שמות יג ,י: וְשָמַרְ תָ אֶת־הַחֻּקָה הַזֹאת לְמוֹעֲדָהּ

The Baal haTurim states that the word "le'mo'adoh" contains a 'Vav' (after the 'Mem'), a hint that one is obligated to wear Tefilin only six days in the week (Tefillin are not worn on Shabbos).

Moreover, he says, there are six Yomim Tovim in the year that we also do not wear them - Pesach, Shavuos, Rosh ha'Shanah, Yom Kippur, Succos and Shemini Atzeres.