Rav Chaim Palagi zt’l (Sefer Tenufah V’Chaim) explains this pasuk by quoting the Gemara (Pesachim 114A) that says that one who often eats delicacies will end up owing money to a lot of people and he will have to sleep in the attic of his home to hide from his creditors. But if one eats vegetables and simple foods, he can sleep out in the courtyard, as he has no one to fear.
Accordingly, the Gemara is saying that one should travel away from ta’avos, and he can then go to the courtyards (chatzeros) and live without fear.