Gemara

13b (Mishnah): Stam hands disqualify Terumah.

14a: This was decreed because hands touch many things.

108b (Beraisa - R. Muna): If a man touches any of the following before washing his hands in the morning, his hand is worthy to be cut off - his nose, mouth, ear, bloodletting wound, Ever, anus, or barrel of beer.

Touching them can cause blindness, deafness, or bad breath.

109a (Beraisa - R. Noson): The Ru'ach (spirit) that dwells on hands before washing is a Bas Chorin (dignified). It goes away only if each hand is washed three times, alternating.

Pesachim 111b: If one does any of the following without washing his hands (afterwards), he will be afraid, and he will not know why he is afraid:

For eating cress, letting blood, and cutting his hair or nails, he will be afraid for 30 days, seven days, three days, or one day, respectively.

Bava Kama 17a (Rabah bar bar Chanah): I asked R. Yochanan a question. He needed to relieve himself. He didn't answer until he washed his hands, put his Tefilin back on and blessed on them.

Sukah 26b (Beraisa): If one forgot, and had relations while wearing Tefilin, he may not hold the strap or box until he washes his hands, for hands touch things.

Nazir 59a - Question (Rav): May one scratch through a garment during prayer (to remove a louse)?

Answer (R. Chiya): No.

The Halachah does not follow R. Chiya.

Rishonim

Rosh (Berachos 9:13): The Yerushalmi says that if one was in a Beis ha'Kisei (privy) or Kaspoklarya (and heard thunder or saw lightning), if he can leave and bless Toch Kedei Dibur, he does so.

Ma'adanei Yom Tov (50): Kaspoklarya is a bathhouse or slaughterhouse.

Rashi (Nazir 59a DH Ika): The Gemara asked whether one who is praying may scratch his garment to make lice fall from it, since he does not touch his flesh. The Gemara concludes that since he does not touch, it is permitted.

R. Bechayei (Bamidbar 19:11): The custom to wash the hands after returning from a Mes commemorates the ashes of Parah Adumah. Also, it hints to Techiyas ha'Mesim. Then, the entire world will be careful about Tum'ah.

Poskim

Shulchan Aruch (OC 4:18): The following require Netilah with water: one who rises from the bed, or leaves the Beis ha'Kisei or bathhouse.

Magen Avraham (18): One who leaves a Beis ha'Kisei must wash immediately, just like (after waking) in the morning.

Gra (DH veha'Yotzei): Bava Kama 17a teaches that one must wash after leaving the Beis ha'Kisei.

Mishnah Berurah (39): He needs to wash three times only for rising from bed. Some are stringent also about marital relations and going between the dead and. Heichal ha'Kodesh is stringent also about leaving a Beis ha'Kisei. The Magen Avraham (7:1) rejected this.

Kaf ha'Chayim (61): Some say that one must wash three times after leaving a Beis ha'Kisei only if it is fixed. Kaf ha'Chayim (Palagi) and Chesed la'Alafim say that one must wash with a Revi'is, and not go four Amos before washing. One should be stringent to wash three times after leaving the Beis ha'Kisei, passing among Mesim or relations. One who is stringent about all of these matters will be blessed.

Kaf ha'Chayim (62): Whenever one must wash three times, it should be alternating.

Mishnah Berurah (40): This is even if he did not eliminate there. Likewise, one must wash after leaving a bathhouse even if he did not bathe there, due to the Ru'ach Ra'ah in these places.

Minhag Yisrael Torah (1:4:11): No one obligates washing three times after leaving a bathhouse.

Shulchan Aruch (ibid.): ... One who cuts his nails or removes his shoes.

Magen Avraham (19): One must wash if he removed his shoes using his hands.

Gra (DH veha'Choletz): We learn from Birkas Kohanim, according to the Rambam, who holds that they wash due to Tum'as Yadayim.

Shulchan Aruch (ibid.): ... One who touches his legs, or scratches his head, or touches a Mes.

Magen Avraham (21): Since we obligate for one who touched a Mes, this connotes that one who merely went near a Mes need not wash. However, the custom is to wash even for entering near a Mes or escorting a Mes.

Shulchan Aruch (ibid.): ... One who deloused his clothing.

Magen Avraham (22): This is even if he did not did not find any lice, and therefore did not touch any.

Shulchan Aruch (ibid.): ... One who had marital relations, or touched a louse, or touched his body with his hand.

Gra (DH veha'Noge'a): The Rosh holds that they enacted to wash in the morning, for Terumah, and all those due to filth, because hands touch things. This is like it says in Sa'if 21.

Mishnah Berurah (46): This refers to filthy places that have Milmulei (dirt due to) sweat.

Shulchan Aruch (ibid.): If he did one of these and did not wash, if he is a Chacham, he will forget his learning. If he is not a Chacham, he will go crazy.

Gra (DH u'Mi): This is from the Beraisa of Ma'aseh Torah of Rebbi. There are three differences.

Damesek Eliezer (28): The Beraisa is brought in Reishis Chachmah Sof Perek Derech Eretz 20. There it says that one who let blood washes his hands in salt, and one must wash for killing (but not merely touching) a louse, or for putting on shoes.

Mishnah Berurah (47): Eliyahu Rabah explains that he is clothed with a Ru'ach of lunacy, which can lead to Aveirah.

Shulchan Aruch (227:3): If one was in a Beis ha'Kisei and heard thunder or saw lightning, if he can leave and bless Toch Kedei Dibur, he does so.

Magen Avraham (2): E.g. if he did not yet touch filth and did not yet eliminate, or if he can wash (and bless within the time).

Eshel Avraham: Even though one who leaves a Beis ha'Kisei must wash even if he did not eliminate, he can bless, since he must wash only due to Ru'ach Ra'ah.

Eliyahu Rabah (227:6): Birkas Avraham says that we discuss one who did not wipe or clean. If his hands were filthy, he would need to wash or clean them first, and he could not bless within Toch Kedei Dibur. Ohr Chadash says that the Shulchan Aruch discusses one who was in a filthy area, and not a Beis ha'Kisei. He overlooked Birkas Avraham and the Magen Avraham. Even though in Siman 4 we always require Netilas Yadayim (after leaving a Beis ha'Kisei, even if he did not eliminate), here, to enable him to bless, we are lenient unless his hands are filthy. Ma'adanei Yom Tov teaches that a bathhouse or slaughterhouse is the same as a Beis ha'Kisei.

R. Akiva Eiger: The Magen Avraham connotes that if he eliminated, even if he did not touch a dirty place he may not bless before washing. In Siman 613 (brought below), it seems that he may bless without washing. Netilas Yadayim is needed only for Tefilah! This requires investigation.

Mishnah Berurah (9): The same applies if he was in a butchery where they cut meat, and it has a very rotten smell, or he was walking in a filthy alley, and he can leave immediately.

Kaf ha'Chayim (15): The same applies if he was in a bathhouse or butchery, which is like a Beis ha'Kisei. If one awoke [in the morning] and heard thunder or saw lightning, if he cannot wash and bless [Toch Kedei Dibur], he thinks the Berachah in his heart. However, based on the custom to bless without Shem and Malchus, he may say the Berachah before washing.

Kaf ha'Chayim (17): The Zohar forbids blessing due to the Ru'ach Ra'ah, until he washes.

Kaf ha'Chayim (4:65): Kerem Shlomo learns from the Magen Avraham (227:2, brought below) that one must wash only if he eliminated in the Beis ha'Kisei, even if it is fixed. Similarly, one must wash after leaving a bathhouse only if he bathed there. He said to see the Pri Megadim. The Eshel Avraham (7:1) initially said so, but concluded that the Magen Avraham always obligates washing after leaving a fixed Beis ha'Kisei, due to Ru'ach Ra'ah. It clings to his hands. The Chida (Bris Olam, on Sefer Chasidim 823) says so in the name of the Zohar. Amudei ha'Shulchan (on Kitzur Shulchan Aruch 2:14) says that the same applies to leaving a bathhouse.

Shulchan Aruch (613:3): If (on Yom Kipur) one urinated and wiped (the drops off his feet with his hand) or eliminated and cleaned himself, he may wash, for his hands are dirty.

Magen Avraham (2): The Bach says that at night, he should wipe or clean in order to avoid the argument [about whether he may wash]. During the day, since we pray the entire day, he need not do so.

R. Akiva Eiger; Seemingly, one who leaves a Beis ha'Kisei needs to wash due to Ru'ach Ra'ah, even if he did not eliminate. This should be permitted even on Yom Kipur! This requires investigation.

Note: Normally, we wash the entire hand. On Yom Kipur, we wash only until the end of the fingers. Migdol Oz (Hilchos Tefilah 7:8) and Kaf ha'Chayim (4:14) explain that this is because Ru'ach Ra'ah has less authority than on a regular day. Perhaps the same applies to the Ru'ach Ra'ah of a Beis ha'Kisei!