The Kabalists answer the basic question of why Hashem created the world with the comforting response of "tevah hatov l'heitiv - the nature of the Good One is to bestow goodness." Rashi (Breishis 2:4) notes that this world was created with the letter hey which has an opening at the bottom, symbolizing man's descending into oblivion if he is not successful in leading a righteous life. Moreover, we are taught that the world to come is symbolized by the letter yud, the smallest letter, proclaiming that those privileged to go there are the minority of the population. If Hashem is good, and He breathes a living soul of His essence into man, literally a chip off the magnificent spiritual block, then why did He make the world so challenging that only a minority end up succeeding and reaching the world to come?
The high failure rate in this world is further emphasized by the Mishnah (Avos perek 5) which teaches that in the ten generations from Adam to Noach the great majority of the world population did not live a noble life. Similarly, from Noach to Avraham there were again ten generations and again man failed to live up to his potential, and Avraham received the reward that was initially allocated to all of them.
The Ohr Ha-Chaim Hakadosh (Breishis 3:4), in beginning to analyze Adam's sin of eating from the eitz ha-daas, similarly asks why Hashem did not diminish the power and attraction of the yetzer harah thus making man's mastery over it easier and more manageable? His answer is that in accordance with the challenge and effort to defeat the yetzer harah is the reward and benefit for both this world and the world to come. He cites the Mishnah (Avos 5:26) which states, "in accordance with the exertion is the reward."
The Ramchal (Da'as Tvunos 18) teaches that the good which Hashem extends to man is the opportunity to connect with and benefit from the Shechina - the Divine. To capitalize on this opportunity one must fulfill the six hundred and eleventh commandment (Devorim 28:9) of "v'holachto biderachav - walking in His ways." Man is to emulate Hashem who is all giving and perform acts of kindness and spirituality.
Were man to receive a reward without performing and accomplishing to earn it, the reward would be "Na'amah d'kisufah - bread of shame", i.e. degrading and debasing. Perhaps this is what our Rabbis are teaching when they state (Shabbos 127a), "receiving guests is greater than greeting the Divine Presence", for it is better to have a relationship with Hashem in a manner of giving, and thereby emulating His exalted character, than to encounter Him by simply being the recipient.
What emerges is the realization that meaning, purpose, and fulfillment in this world are, as stated by the introductory words of Mesilas Yesharim, comprised of overcoming the obstacles and challenges of the evil inclination. Freud taught that man is inherently bad and possesses an id which yearns for negativity; Judaism says man possesses a yid - a holy Divine image that enables him to overcome his desire for bad. Man is to emulate Hashem, and as His nature is to do good so too must man use his free will to overcome the desire to do bad and do good instead.
Our initial question assumed that tests and challenges are not really good for man. However, the Ramban in his commentary on the akeida (Breishis 22:1) teaches that Hashem only tests those that can pass the test. Avraham became the great father of our nation because he was tested with ten tests through which he became elevated and actualized his potential. Each individual as well is to be cognizant of the fact that they possess a unique mission and potential and a unique array of challenges. Our challenges are Hashem's way of offering us His l'heitiv - His ultimate goodness, which is the opportunity to grow and earn the best of this world and the next. Indeed, the Ohr Ha-chaim cited earlier ends his treatment of this most important concept with the words "praiseworthy is the people for whom this is so". It is all a matter of perspective.