I
"These are the descendants (toldos) of Esav" (Breishis 36:1). The word "toldos" is spelled in four different ways: with two vavs, no vavs, only the first vav, or, as in our pasuk, only the second vav.
Rav Shimshon Raphael Hirsch (25:12) explains that a missing vav of the root, the first vav, indicates an internal deficiency in the descendant. A missing vav of the plural, the second vav, represents a lack in the number of descendants.
When describing Hashem's creations - "These are the products (toldos) of the heavens and the earth" (Breishis 2:4) both vavs appear, since Hashem's works lack neither quality nor quantity. Similarly, the description of the lineage of Dovid Hamelech and the Mashiach - "these are the generations (toldos) of Peretz" (Rus 4:18) - uses two vavs to reflect their completeness.
The descendants of Esav are great in number but lacking spiritually and morally. Therefore "toldos Esav" (Breishis 36:1) omits the first vav but contains the second.
Some of the descendants of Yitzchak, i.e. Yaakov and his sons, were spiritually great. Hence, "these are the offspring (toldos) of Yitzchak" (Breishis 25:19) contains the first vav. Since Esav and his children lacked spiritual greatness, the second vav, which would indicate greatness among all of Yitzchak's descendants, is omitted. Rashi's comment on the word toldos - "Yaakov and Esav who are spoken of in the parsha" - may reflect the presence of the first vav and the absence of the second vav, respectively.
"The descendants (toldos) of Yishmael" (Breishis 25:12) is spelled without either vav. Rav Hirsch explained that this is due to the fact they were neither spiritually great nor numerous. This requires explanation, as Yishmael had many sons and innumerable descendants! The Kli Yakar (Breishis 25:23) states that there are gerei tzedek from Esav but not from Yishmael (see Chasam Sofer Kesubos 53a). As such, perhaps Rav Hirsch is saying that quantity is positive only when at least some quality results.
II
Surprisingly, the first vav is omitted when describing Yaakov's sons, "These are the descendants (toldos) of Yaakov" (37:2). Rav Hirsch explains that this indicates the moral deficiency of Yaakov's sons in their sin against Yosef. This paradigmatic interpersonal sin (see Meshech Chochma Vayikra 16:30) can be partially explained by the family history. Avraham had a bad son, Yishmael, and Yitzchak had a bad son, Esav. The defect in the offspring of Yitzchak is rooted in the previous generation: Avraham begat Yitzchak. Since Avraham had a Yishmael, Yitzchak had an Esav. Yaakov's sons anticipated that their generation would be no different than the previous ones, and thus were expecting there to be a bad son among Yaakov's children. When Yosef behaved inappropriately (Rashi 37:2), they jumped to the erroneous conclusion that he was the Yishmael or Esav of their generation. They misunderstood his dreams as confirmation of this hypothesis and this led them to their terrible sin (see Malbim Breishis 37:4).
In fact, Yosef was a righteous person whose behavior was somewhat different from his brothers'. The tragic mistake of treating someone whose path in the service of Hashem is different from one's own as a wicked or heretical person is precisely the sin which caused the destruction of the second Beis Hamikdash. Substantive but acceptable differences led some to consider their coreligionists to be tz'dukim, beyond the pale. Perushim, righteous Jews, pursued each other based on the false imagination of the other as a heretic (Netziv, Meishiv Davar 1:44).
Just as Yishmael was thrown out of Avraham's home with Hashem's consent (Breishis 21:10-12), Esav was distanced by Yaakov (Breishis 33:13-14, see Rashi). Later, idolaters were purged by Moshe Rabbeinu (Shemos 32:27-9). It is critical, however, not to extrapolate from these precedents to cases that lie beyond certain rigorous borders. It is proper to distance oneself from heretics, pray for their downfall (v'lamalshinim - v'chol haminim), and even, when possible, pursue them (Avodah Zarah 26b). However, the mistake of Yosef's brothers and the Perushim of bayis Sheni was to attack righteous individuals whose ways differed from theirs.
Rav Hirsch himself was famous for Austritt, i.e. stepping away from the organized community which was controlled by heretical Reform Jews. Other great rabbonim disagreed. In any event, each case is somewhat unique and must be individually analyzed (see Kovetz Igros Achiezer vol. 1, p. 243-244).
While the aforementioned Netziv bemoaned unnecessary disunity in Klal Yisrael, others considered the disunity necessary and critical in order to maintain ideological purity. Now, over a century later, these disputes continue. To what extent should Torah-true Jews separate themselves from heretics? What about their innocent children, whom the Rambam (Hilchos Mamrim 3:3) requires us to "pull close with words of peace until they return"? How does one deal with those who subscribe to the fundamentals of our faith (ikarei emunah), but view the halachic process in a radically different way? Should they be attacked, ignored, or embraced? Some otherwise Orthodox Jews have succumbed to the temptations of promiscuity or alternate lifestyles. Should one express outrage or sympathy? Might it depend on whether the behavior is recognized as sin or trumpeted as perfectly acceptable?
As we read parshiyos Vayishlach and Vayeshev, we must learn the lessons of the four spellings of toldos and attempt to properly balance the beloved ideals of truth and peace. This balance will lead to our ultimate redemption (Zecharyah 8:19).