Gemara

(Rav Chana bar Bizna): Any fast that does not include sinners is not a proper fast;

We learn from Chelbenah. It smells bad, yet it is included with the spices for the Ketores.

(Abaye): We learn from "va'Agudaso Al Eretz Yesadah."

Sanhedrin 44a (R. Aba bar Zavda): (After Achan sinned, Hash-m said) "Chata Yisrael." Even if a Yisrael sinned, he is still called a Yisrael.

74a (Rav Yanai bar Aba): Kidush Hash-m applies only in front of 10 Yisre'elim. It says here "v'Nikdashti b'Soch Bnei Yisrael", like it says about the Meraglim "Hibadlu mi'Toch ha'Edah ha'Zos", and there were 10 (evil) Yisraelim there.

Rishonim

Rambam (Hilchos Talmud Torah 7:4): We do not count a Menudah (someone excommunicated) for a Zimun or for 10 for anything that requires 10.

Rivash (172, citing in Beis Yosef OC 55 DH Kasuv): The Rambam says so only if he was excommunicated. Even though the Gemara (Mo'ed Katan 15a) did not list this among laws of Niduy, the word Niduy itself means isolation and separation. If one transgressed and was not excommunicated, we do not find that he does not join. Maharam me'Rotenburg explicitly says that one who transgressed a decree of the Tzibur joins until he is excommunicated.

Mordechai (Yoma 725): (On the night of Yom Kipur) we enter the Beis ha'Keneses and permit the Cherem against praying with one who transgressed a decree of the Tzibur, even if he does not request to be permitted, for any fast that does not include sinners is not a proper fast.

Poskim

Shulchan Aruch (OC 619:1): On the night of Yom Kipur, the custom is that the Shali'ach Tzibur says 'in the Yeshiva above, and in the Yeshiva below, Al Da'as (with the consent of, or according to the will of) Hash-m and Al Da'as the congregation, we permit to pray with the sinners.'

Beis Yosef (DH Arvis): This text is based on Kol Bo (68, 33b citing Maharam) and Tashbatz (131).

Bach (DH u'Mah): Why do we first say 'in the Yeshiva above', and afterwards 'we permit'? If 'we' includes the Yeshiva above, what is the meaning of 'in the Yeshiva'? Also, it is difficult to say that everyone includes himself with 'the Yeshiva'. I say that it should say 'in the Yeshiva above, and in the Yeshiva below, (they) permit to pray with the sinners, Al Da'as Hash-m and Al Da'as the Tzibur. Also Tzedah la'Derech says so. We say this only lest people be astounded (that we pray with the sinners), so we inform them that also the Yeshiva above consented, based on the Drashah in Kerisus. The Yeshiva below, wherever it is fixed, decreed and also permitted to pray with the sinners on Yom Kipur. We say 'Al Da'as Hash-m and Al Da'as the Tzibur', lest they think that since they may pray with us, they are no longer considered sinners. The Heter is only Al Da'as Hash-m, that they repent and not return to sin. Since we merely inform people, we do not need two to say it. Mahara Tirana says so. It suffices for the greatest person to say it. Maharil (Hilchos Yom Kipur 5) says so.

Taz (2): We permit to pray with the sinners. This suffices to show that only this is permitted (but they are still considered sinners)! Also, the Bach's text alludes to distancing them if they do not repent. Hash-m knows whether they will repent. (If they will not,) this prosecutes! Rather, three men are in place of the Tzibur, and they join Hash-m to permit. The Shulchan Aruch's text is correct. First we mention the Yeshivos, for they decided to join Hash-m to permit.

Taz (1): Minhagim requires another two. The Shulchan Aruch did not say so. I say that if the Shali'ach Tzibur says it, we do not need two others, for he is the Shali'ach of the Tzibur, so it is as if they say it. If the Rav or the greatest person of the Tzibur says it, we need another two, to be in place of the Tzibur. The Maharil used to say this by himself. He was the Shali'ach Tzibur. The Levush says that two important people of the Tzibur stand near the Shali'ach Tzibur, so they will be three to permit praying with the sinners, but we do not need three for Kol Nidrei, which is only a stipulation that future vows be invalid. If so, the two should go away before Kol Nidrei! I say that we want publicity that the entire Tzibur consents, therefore they must remain until the end of Kol Nidrei.

Mishnah Berurah (1): Nowadays the greatest of the city says it. He must join two others. They join Hash-m to the Tzibur's intent to permit the sinners from the status of sinners (i.e. one who transgressed a decree of the Tzibur or an Aveirah). If one was excommunicated, he may not join for any Devar shebi'Kedushah (something that may be said only amidst 10 men). They can stipulate to be stringent not to pray with him in the Beis ha'Keneses even if there are 10 (Tzadikim). Perhaps this was decreed! Therefore, we permit them so we can pray with them, since a fast without sinners is not a fast.

Shulchan Aruch (55:11): If one transgressed a decree of the Tzibur or an Aveirah, if he was not excommunicated, he counts for a Minyan of 10.

Beis Yosef (DH Kasuv): Ha'Manhig writes that if a transgressor was not excommunicated, he counts for a Minyan and he is obligated in all the Mitzvos. We learn from Achan that even if a Yisrael sinned, he is still a Yisrael. However, if he was excommunicated, he does not join. If not, in what respect did they separated him from them?! In a Teshuvah, the Rashba said that if one was excommunicated for an Aveirah, he does not join. This is not due to the Cherem, for if so, it would apply even to Niduy for monetary reasons! Rather, it is for becoming a transgressor through a severe transgression for which he was excommunicated.

Beis Yosef (ibid.): In another Teshuvah (371), the Rivash said that a Menudah does not count towards 10, but the Gemara does not forbid him to pray with the Tzibur, as long as he is not within four Amos of others. If the Tzibur accepted to be more stringent, they must do like they accepted. If the Rav sees a need to be extra stringent, he may.

Gra (DH Avaryan): We learn from Sanhedrin that he retains his Kedushah. Even though he is disqualified for testimony, women (even though they cannot testify) would join for a Zimun if not for lewdness. Kusim (they converted due to fear of lions, but they did not accept the Oral tradition) join, even though they transgress mid'Rabanan laws. One who ate Tevel does not join because this is not Keva (being fixed to eat together). Also, he can join at a different meal (when he ate Heter, even though he has sinned).

Mishnah Berurah (46): The Pri Megadim says that this is only if he transgressed for pleasure, but if he transgressed to anger Hash-m even in one matter, or if he is a Mumar to serve idolatry or desecrate Shabbos in public, he is like a Nochri and he does not join.

Mishnah Berurah (47): This is even for an Aveirah of which one is Chayav Misah. We learn from Achan. Even though he sinned, he keeps his Kedushah. Kara'ites, or anyone who denies Oral Torah, does not join for any matter of Kedushah.

Kaf ha'Chayim (62): The Radvaz (415) says that he joins. Tashbatz says that if he does not want to fulfill Tefilah, he is a Rasha and he does not join. I.e. all the more so one who does not want to fulfill a Torah Mitzvah does not join. This is primary.

Melamed Leho'il (1:29): A person is Mechalel Shabbos, not only for his work, but also for incense. He does not even say Kidush or Havdalah. The Eshel Avraham (55:4) and Chacham Tzvi (38) say that he does not join to a Minyan. She'elas Ya'avetz (1:49) connotes that as long as he was not excommunicated, he joins, he did not explicitly say so about one who is Mechalel Shabbos b'Farhesya (in public). Meshiv Davar (9) forbids. The Magen Avraham (199:2) says that he does not join for a Zimun. Rashban (90) brings from Doresh l'Tziyon says that this is only if he transgresses (Shabbos) Lehach'is (to anger Hash-m). Nowadays, people do not transgress Lehach'is. This is astounding. We do not join one who transgresses any Aveirah Lehach'is! Rashban also brought nowadays that strongarms prevail, if such a Kohen ascends to Duchan (give Birkas Kohanim), we cannot make him descend. Yehudah Ya'aleh (YD 50) says that nowadays, we permit wine touched by Mechalelei Shabbos, until people testify in Beis Din that they transgressed in front of 10 Yisre'elim. Many Acharonim hold that it is only mid'Rabanan that a Mumar to be Mechalel Shabbos is a Mumar to the entire Torah. In summary, he should not join, but the custom is to be lenient is Ungarn, and all the more so in Ashkenaz. A case occurred in which a man used to keep his store open on Shabbos. He became an Avel, and the Gabai did not let him be Shali'ach Tzibur. A Gabai in another Beis ha'Keneses permitted him, for this is the custom. Presumably it is based on Binyan Tziyon (Chadashos 23), who says that people who are Mechalel Shabbos nowadays are like a Tinok she'Nishba (a baby captured and raised by Nochrim), for most Yisre'elim in our land are Mechalel Shabbos. They do not intend to deny our faith. Sho'el u'Meshiv said so about Mechalelei Shabbos from America. R. Akiva Eiger (YD 264) wrote similarly. One who relies on them to join such people to a Minyan has whom to rely on. If one can go to another Beis ha'Keneses without embarrassing anyone, it is better not to rely on this leniency. Also, since most people are Mechalel Shabbos, this is not considered a severe Aveirah, so doing so in public is like doing so in private.

Igros Moshe (OC 1:23 and 2:19): A verse teaches that one must be Moser Nefesh (forfeit his life) and not transgress (when Nochrim seek to make Yisre'elim transgress) when 10 Yisre'elim are present. The same verse teaches about Devarim shebi'Kedushah. Just like the former applies even in front of deniers and apostates, for they are still Yisre'elim, also the latter. However, some require that the majority be Kosher. Tosfos holds that a Mumar to eat Nevelah is not considered someone to whom Shechitah applies. One who does not wear Tefilin is like one who is not commanded; he is Pasul to write them (Gitin 45b). Perhaps if one prays now, Tosfos would agree that he joins. In pressed circumstances one may join him for Kaddish, Kedushah and Borchu, but it is not Tefilah b'Tzibur. The same applies to joining him for Kri'as ha'Torah. Zecher Yitzchak says that a Mechalel Shabbos does not join. Perhaps he refers only to Tefilah b'Tzibur. He says that R. Yosef Razin says that they do not join for Musaf, since we do not accept Korbanos from Mumarim. This shows that for they join for (other) Devarim shebi'Kedushah.

Minchas Yitzchak (3:26): One may join a Mechalel Shabbos only in greatly pressed circumstances. If one would not be Mechalel Shabbos in front of the Nasi, he is not considered Mechalel Shabbos b'Farhesya (Eruvin 69a). However, Darchei Teshuvah (YD 119:34) says that this is only for other Isurim, but not for Isurei Shabbos.

Chazon Ish (YD 2:28): Ahavas Chesed (of the Chafetz Chaim) brings from Teshuvah Maharam Lublin that since nowadays we do not know how to rebuke (Erchin 16b), after rebuke is like before rebuke (so no one is a Rasha whom we may hate). They are like a Tinok she'Nishba.

Yabi'a Omer (7 OC 15): Based on these Poskim, if a Kohen is Mechalel Shabbos b'Farhesya but did not totally cast off the yoke of Mitzvos, we may let him Duchan. The Rambam (Ma'amar Kidush Hash-m) says that we should not distance such people. We should encourage them to do Mitzvos, especially nowadays that we cannot fulfill everything properly. This has helped many to do full Teshuvah. If also others Duchan, we do not discourage him, but if he is the only Kohen, we should encourage him to leave the Beis ha'Keneses. If we cannot, he may Duchan.