There is a fundamental principle in Avodas Hashem discussed extensively in the seforim. Our teachers repeatedly remind us that when we encounter any kind of struggle, our Avodas Hashem is way more cherished on High.
The pasuk says in Shir Hashirim, "מֵשִׁׂגִיחַ מִן הַחַלֹּנוֹת מֵצִיץ מִן הַחַרַכִּים," [Hashem, at the time of Yetzias Mitzrayim, was] observing through windows, peering through the crevices. (שׁה"ש ב ט)
Rav Sa'adya Gaon translates this pasuk as an analogy to describe two primary methods of Divine Conduct. At times, Hashem's Presence is clearly visible and 'palpable'. This type of Conduct is alluded to in the words "מֵשִׁׂגִיחַ מִן הַחַלֹּנוֹת," He observes through the windows, from which one can see and be seen clearly. Yet, there are times that Hashem conceals His Control from Mankind.
Yet, even in moments where all seems lost and the situation feels hopeless, Hashem is still there watching over us from a hidden place. Periods like these are referred to in the pasuk as "מֵצִיץ מִן הַחַרַכִּים," He peeks from the crevices. Even at a time when the Divine Command is not eminently visible, Hashem is still 'there' looking out for His beloved children, ensuring that no harm or evil befalls them.
This is what Avraham was asking. Avraham Avinu was pondering why Hashem was concealing Himself from the world, seemingly neglecting His Creations and allowing evil to reign. To this Hashem replied: "I am the One in charge of the world", saying that this "concealed state" is orchestrated with plan and purpose; Hashem, in His wonderous Ways, was hiding His governance from This World so that people should seek Him, in spite of their hardships and the difficulty it may entail.
Hence, while it seemed that Hashem had forsaken This World, He was still there, albeit hidden from Man's conscience since He was waiting for someone to reach out to Him. Indeed, Avraham ultimately understood the message being conveyed to Him and reached out to Hashem despite the dominant godless environment that surrounded him.
This is perhaps what the Midrash is alluding to by saying "הֵצִיץ עָלָיו בַּעַל הַבִּירָה", the man in charge of the palace "peered". Since, as explained, the term הֵצִיץ generally refers to peeking from a small opening, unnoticeable to the subjects being observed.
When Hashem runs the world with His "hidden ways", it is to give Man an opportunity to reach out to Him, and to benefit eventually from the tremendous reward bestowed upon the individual who overcomes the challenges and obstacles that stand in the way.
Pleasant Offerings
When Noach left the teiva after the mabul he erected a mizbeach and sacrificed korbanos for Hashem. The Torah tells us that Hashem accepted the korbanos and thereafter promised that He would no longer bring a mabul to destroy the world. וַיָּרַח ה' אֶת רֵיחַ הַנִּיחֹחַ, וַיֹּאמֶר ה' אֶל לִבּוֹ לֹא אֹסִף לְקַלֵּל עוֹד אֶת הָאִדָמָה בַּעֲבוּר הָאָדָם, כִּי יֵצֶר לֵב הָאָדָם רַע מִנְּעֻרָיו... - and Hashem smelled the pleasant aroma, and Hashem said in His heart "I will not again curse the ground for man's sake; for the inclination of man's heart is evil from his youth.
The Kedushas Levi explains this verse with an idea from his father: Hashem proclaimed that He will no longer annihilate humanity precisely because they possess this tendency towards wickedness. When man overcomes this powerful yetzer hora and serves Hashem despite his surroundings, he brings Hashem immense pleasure. The beginning of the posuk connects well with the end the רֵיחַ הַנִּיחֹחַ is davka because of the יֵצֶר לֵב הָאָדָם רַע...
(קדושת לוי פר' נח ד"ה וירח)
The Unknown Destination
The Sefas Emess asks, why Hashem gave Avraham Avinu such a vague instruction of "לֶךְ לְךָ... אֶל הָאָרֶץ אֲשֶׁר אַרְאֶךָּ", go to the land that I will show you. Why did Hashem not instruct him immediately to go to the land of Canaan?
This can be understood based on the above-mentioned קְדֻשַׁת לֵוִי. Hashem wanted to show us that although one who serves Hashem when everything is smooth and easy, brings great joy and nachas ruach on High, Hashem derives even greater pleasure, k'vayachol, when a person yearns to connect with Him in difficult times when Hashem's Presence is concealed in This World.