Gemara

(Beraisa): "V'Dibarta Bam" - in Shma you may speak (out loud), but not in Tefilah (Shemoneh Esre);

Tosfos Yeshanim (DH v'Dibarta): Alternatively, while saying Shma you may return Shalom to someone you must honor, but not during Tefilah.

"Bam" you may speak, but not in other matters.

Rashi (DH Bam): You may speak Divrei Torah, but not frivolous matters.

R. Chananel (DH Tanu Rabanan): You may interrupt Divrei Torah for Shma, but not for other matters.

Rav Acha says "V'Dibarta Bam" - make Divrei Torah Keva, not Arai.

Rava: One who speaks Sichas Chulin transgresses an Aseh - "V'Dibarta Bam", speak Divrei Torah but not other matters.

Rav Acha bar Yakov says, he transgresses a Lav - "Kol ha'Devarim Yege'im Lo Yuchal Ish Ledaber".

Avos 1:17: Rebbi's son Shimon says, I grew up among Chachamim and I did not find anything better for the body than to be silent. Anyone who talks too much brings sin.

Sukah 28a: R. Eliezer and R. Yochanan ben Zakai never spoke Sichas Chulin.

Shabbos 113a: "V'Daber Davar" - your speech on Shabbos should not be like your speech on a weekday.

Rishonim

Rambam (Hilchos De'os 2:4): One should usually be silent and speak only matters of Chachmah or things essential for his health. Rav never spoke idle talk in his life. The conversation of most people is idle. Even regarding physical needs one should not speak too much. Chachamim taught that anyone who talks too much brings sin, and that the best thing for the body is to be silent.

Rambam (Hilchos Shabbos 24:4) and Rosh (Shabbos 15:2): One may not speech much idle talk on Shabbos because of "V'Daber Davar", your speech on Shabbos should not be like that of Chol.

Source (Rosh, ibid.): When R. Shimon's mother spoke too much on Shabbos, he would tell her 'it is Shabbos' and she would be silent (Vayikra Rabah Sof Perek 34). Greeting people on Shabbos was permitted with difficulty (Yerushalmi).

Perush ha'Mishnayos (Avos 1:17): There are five categories of speech:

Learning Torah is a Mitzvah equal to all the Mitzvos;

Forbidden speech such as false testimony, lying, Lashon ha'Ra...

Despised speech that has no benefit. This is the conversation of most people, e.g. events that happened, the king's mannerisms, his palace, how Ploni died or became rich...This is idle talk. Chachamim strove to avoid it. Denouncing attributes and praising lowly things is also despised.

Cherished speech is praising attributes and denouncing lowly things; stories and songs that inspire virtuous behavior, praising important people and publicizing their attributes so people will follow their ways, and denouncing Resha'im to distance people from their ways. Learning these things is called Derech Eretz.

Permitted talk is what pertains to a person's business, food, drink and other needs. It is praiseworthy to minimize such speech. It would be ideal if a person would constantly speak what is a Mitzvah or cherished. However, he must beware that his deeds fit his words, and to say things concisely.

Sha'arei Teshuvah (of R. Yonah, 3:6): One should not hear idle words, for the ears are scalded before all other limbs (Kesuvos 5b).

Me'iri (DH l'Olam): One should be very careful not to speak idle words, only what is necessary. Chachamim figuratively said that one who speaks idle words transgresses a Lav.

Orchos Tzadikim (Sha'ar 26, the 13th step of Teshuvah): Aveiros that are light in one's eyes should be severe to him, e.g. idle words.

Poskim

Shulchan Aruch (OC 151:1): We do not act frivolously in a Beis ha'Keneses or Beis Medrash, e.g. jesting or idle talk.

Eshel Avraham (1): This forbids even Sichas Chulin that is not idle, e.g. for the needs of one's income, which is permitted outside. Idle talk anywhere is forbidden by an Aseh, at least mid'Rabanan.

Shulchan Aruch (307:1): Your speech on Shabbos should be unlike your weekday speech; you should not discuss Melachah or business. One may not even speak much idle talk.

Gra (DH Leharbos): Idle talk is forbidden on Shabbos only if it is excessive. R. Shimon did not comment unless his mother spoke much. Talking about matters that are no longer relevant is permitted (307:6), even though the Rambam calls this idle talk.

Note: In Even Shleimah (10:11), the Gra says that the sixth judgment a person undergoes after death is Kaf ha'Kela. He is slung from one end of the world to the other for idle words. The punishment for forbidden words, e.g. scoffery, vows, arguments...especially on Shabbos or Yom Tov, is impossible to estimate (Alim li'Trufah, cited in footnote 9).

Rema: If people enjoy discussing events or news it is permitted like during the week. One who does not enjoy them may not say them for the sake of another who enjoys them.

Beis Yosef (DH v'Chasuv): Stories of kings and wars for those who enjoy them is permitted, just like youths who enjoy running and jumping may do so.

Mishnah Berurah (4,5): One should not spend much time discussing them. They are less Oneg than eating and drinking, and for those who do not learn Torah during the week Shabbos is primarily for Torah. Even Chachamim who learn all week should not be overly Batel from Torah.

Mishnah Berurah (37:7): One must be careful not to speak idle words while wearing Tefilin. In addition to the Isur of idle words, he may come to Hesech ha'Da'as from Tefilin.

Mishnah Berurah (156, b'Sof): One who speaks Sichas Chulin, i.e. disgraceful matters and frivolity, transgresses an Aseh, "V'Dibarta Bam". This is a Lav inferred from an Aseh, which is like an Aseh.

Shulchan Aruch (YD 246:25): If one could engage in Torah and does not, this is "Ki Devar Hash-m Bozah".

Rema: It is forbidden to speak Sichas Chulin.

Aruch ha'Shulchan (35): One should not speak much unnecessary Sichas Chulin.