The Gemara explains that they wore the Bigdei Kehunah during the Payis in order that they should not come to do the Avodah without the Begadim. We could say, therefore, that the Begadim are necessary for the Avodah and do not present a problem of Kil'ayim. (See Beis ha'Levi 1:2:14.)

The Gemara on Daf 69a discusses the question of wearing the Bigdei Kehunah at times when the Kohanim are not doing the Avodah. The Gemara brings a Beraisa that states that in the Beis ha'Mikdash (Tosfos Yeshanim explain that here this refers to the whole of Har ha'Bayis) it is permissible for the Kohanim to wear the Begadim even when they are not involved in the Avodah. Rashi explains that this is because the Bigdei Kehunah were hard and did not provide warmth to the wearer.

Other Rishonim reject Rashi's reason. Tosfos Yeshanim in Yoma says that the Torah was Matir the Isur Kil'ayim at the time of the Avodah and this Heter applies even before or after the Avodah as long as the Kohanim were in the Mikdash, since it would be unreasonable to expect the Kohanim to remove their clothes the second they finished the Avodah. (See also Sha'agas Ari'eih 29 (end) and Mikdash David 36:3.)

However, Tosfos Rid says that the Beraisa that is Matir Bigdei Kehunah in the Mikdash must go according to the opinion that there was no Kil'ayim in the Bigdei Kehunah. Perhaps according to the Tosfos Rid our Gemara also holds like that, unless we say the answer suggested above.

The Beis ha'Levi (1:1:7) suggests that at the time of the Payis the Kohanim wore their Begadim except for the Avnet.

See also Rambam and Ravad, Kil'ayim (10:32) and Klei ha'Mikdash (8:11-12) and Kuntras Maran Riz ha'Levi Yoma page 23).

Dov Freedman