Gemara

(Rav, R. Chanina...): If one could protest against the evil deeds of his household [and it would help] but he does not, he is punished for their sins.

55a (Rav Acha b'Rebbi Chanina): Hash-m told Gavri'el to write the letter 'Tav' in ink on the foreheads of Tzadikim in Yerushalayim and a 'Tav' in blood on the foreheads of Resha'im, so that angels of destruction will kill only the Resha'im.

Midas ha'Din: Why will these be spared and these will be killed?

Hash-m: These are absolute Tzadikim, these are absolute Resha'im!

Midas ha'Din: The Tzadikim could have protested! You know that it would not have helped, but they did not know that!

148b and Beitzah 30a - Question (Rava bar Rav Chanan - Mishnah): On Yom Tov we may not clap hands on each other or over the heart, or dance [lest one fix a musical instrument].

People do this, and we do not protest!

Counter-question (Abaye): Rava taught that one should not sit even with the Lechi (a post in an alley). People sit there, and we do not protest!

Answer to both questions (Abaye): We are silent because it is better that Yisraelim transgress b'Shogeg than b'Mezid!

Assumption: This applies only to mid'Rabanan laws. For Torah laws, we protest even if they will not listen.

Rejection: Mid'Oraisa, one must add to Inuy (begin fasting before Yom Kipur), yet women eat until night, and we do not protest!

Yevamos 65b (R. Ilai): Just like it is a Mitzvah to say something that will be heeded, it is a Mitzvah not to say something that will not be heeded.

(R. Aba): This is an obligation - "don't rebuke a scoffer, lest he hate you; rebuke a Chacham, and he will love you".

Rishonim

Rif and Rosh (Yevamos 21b and 6:21): Just like it is a Mitzvah to say something that will be heeded, it is a Mitzvah not to say something that will not be heeded. "Don't rebuke a scoffer, lest he hate you."

Rif and Rosh (Beitzah 16b and 4:2): We do not protest against people who clap hands or dance on Yom Tov, for it is better that Yisraelim transgress b'Shogeg than b'Mezid. This is only if we are sure that they will not listen. If perhaps they will listen, we protest. This applies even to a Torah Isur. Mid'Oraisa, one must add to Inuy, yet some eat until night, and we do not protest!

Ran (DH v'Hach): Shabbos 148b concludes that we protest about Torah Isurim. The Halachah follows our Sugya (in Beitzah), because actions were done and Chachamim did not protest.

Note: Our text in Shabbos also concludes that we do not protest even for Torah laws.

Rosh (ibid.): Ba'al ha'Itur says that we do not protest about a Torah law that is expounded. However, if something is explicit in the Torah we protest and punish until people stop transgressing.

Korban Nesan'el (2): Sefer Chasidim (262) says that even for a Safek Isur Torah we do not protest [unless it is in the Beis ha'Keneses]. Perhaps the people ate during Bein ha'Shemashos.

Birkei Yosef (OC 608:4): Yashresh Yakov asked from 55a, which says that the Tzadikim were liable only because they did not know that rebuke would not help. Ba'al ha'Itur says that one must protest in any case! He answers that even so, they would not be Chayav Misah [if they knew that it will not help]. Also Semag says so. I say that Ba'al ha'Itur obligates protesting even if he knows that they will not accept it, when we overpower the transgressors. For a Torah Isur, we punish them until they refrain. If we do not overpower the transgressors, and we know that they will not accept it, one need not protest, for it will not help. In such a case, it is better not to say a matter that will not be heard (Yevamos 65b). Kitzur Piskei ha'Rosh in Yevamos brings this, and in Beitzah he brings Ba'al ha'Itur. (There is no contradiction, like I explained.)

Rambam (Hilchos Shevisas Asor 1:7): We do not protest against women who eat and drink until night, for they do not know that it is a Mitzvah to add from Chol onto Kodesh. It is better that they transgress b'Shogeg than b'Mezid. It is impossible to have an officer in every house to warn the women. The same applies to every similar case.

Magid Mishneh: 'Every similar case' refers to things not explicit in the Torah that people are used to being lenient about, and everyone is responsible for himself.

Ritva (Makos 20b DH v'Kamah): We say that it is better that Yisrael transgress b'Shogeg than b'Mezid only regarding an Isur Aseh.

Tosfos (55a DH v'Af): It is proper to rebuke when there is a Safek whether his words will be accepted. If surely his words will not be accepted, it is better that Yisraelim transgress b'Shogeg than b'Mezid, like we say on 148b and in Beitzah.

Poskim

Shulchan Aruch (OC 608:2): Some women eat and drink until dark, for they do not know that it is a Mitzvah to add from Chol onto Kodesh. We do not protest against them, lest they transgress b'Mezid.

Rema (OC 608:2): Regarding every Isur not explicit in the Torah, it is better that people transgress b'Shogeg than b'Mezid. If the Torah explicitly forbids something, we protest.

Shoham v'Yashfe (brought in Likutim, Frankel Rambam De'os 6:7): The Rema rules like the Ritva, who distinguishes Shogeg from Mezid. However, Tosfos (55a) holds that even for Mezid, if one knows that protest will not help it is better that the person transgress without warning than with warning.

Birkei Yosef (3): Divrei Emes proved from the Rosh, Ran, Magid Mishneh and Maharam Alashkar (35) that anything expounded through the 13 Midos is considered mid'Rabanan, like we say on 148b. This is difficult. The Ran (16b DH Hach) explicitly said that the Sugyos argue with each other, and he favored the Sugya in Beitzah over the Sugya in Shabbos! The Rosh and Magid Mishneh did not say anything (about this). Maharam Alashkar does not mean that it is considered mid'Rabanan, rather, that it is not evident to all. If it were evident, even though it is from a Drashah, we would protest.

Mishnah Berurah (3): If perhaps people will listen, we protest even about an Isur mid'Rabanan.

Kaf ha'Chayim (8): Piskei Tosfos (Beitzah 109) obligates rebuking only when one knows that his words will be heeded. However, the vast majority of Poskim obligate even if it is a Safek.

Note: It is hard to say that we know that women would not listen. Perhaps even if most would listen, we know that some would not listen, and it is better that all transgress b'Shogeg rather than some transgress b'Mezid. Chachamim uprooted the Mitzvos of Shofar and Lulav on Shabbos from all of Yisrael, lest an individual transgress Hotza'ah.

Kaf ha'Chayim (10,11): Nowadays women fulfill Tosefes Inuy like men. However, regarding things that people treat lightly and are unlikely to heed us, it is better that they be Shogeg. If people are accustomed to an improper behavior in public, this is considered clear that they will not accept rebuke. One need not protest unless it is explicit in the Torah.

Birkei Yosef (ibid.): (Vayikra Rabah, Reish Parshas Tzav, brings that Aharon squashed the Egel to show people that it has no power, to try to dissuade them from serving it.) Moshe criticized him, and Hash-m was angry with Aharon and wanted to erase him from the Sefer Torah, for it is it is better that Yisraelim transgress b'Shogeg than b'Mezid. (Etz Yosef, Matanos Kehunah - had they believed that the Egel has power, they would be like Shogeg for desiring to worship it. Now that they saw that it is powerless, they were Mezid.) Mahari Ze'evi answered like I said (it is better that they transgress b'Shogeg , for Aharon was unable to stop them.)

Ginas Veradim (1:12): Even without Aharon, Yisrael knew that idolatry has no power. They served it merely to openly permit to themselves Arayos (Sanhedrin 63b). Hash-m was angry at Aharon because he caused that future generations would judge Bnei Yisrael to be Mezid, and add to their guilt. Aharon's generation had no grievance against him. They admitted that in truth, he did not do anything to them.

Note: He must explain "you brought on this nation a great sin" to mean 'you caused that people will consider it to be a great sin." Why did Hash-m punish Aharon due to what future generations will think? Aharon was trying to stop them from serving it! Hash-m could have commanded not to discuss what Aharon did, so future generations will not know that they were Mezid! Chachamim allowed mocking idolatry, without concern for making people Mezid. Eliyahu ha'Navi publicly did so on Har ha'Karmel. Perhaps he was confident that they would cease idolatry afterwards. However, he prayed that they not say that he used witchcraft. (I.e. he was concerned for this.) It seems that Izevel continued serving idolatry after this episode, and afterwards Yehu gathered a whole building of worshippers of Ba'al.

Birkei Yosef: The Rosh (Teshuvah 6:3) did not mention the Ba'al ha'Itur's opinion. We can say that he knew that the questioner had no power to force people to fulfill the Halachah.