The Gemara says that according to the opinion that the lottery was in holy clothing that a cohen who "lost" the lottery would change back to regular clothing.
Why would he have to do that? The lotteries refer only to the public sacrifices as the Tamid. I assume that all the cohanim were busy all day long bringing private sacrifices.
Eli Turkel
All the four lotteries that were performed in the Bais Hamikdosh took place
in the morning. Any Kohen not selected in one of these lotteries, was not
guaranteed to have a job to do during the course of that day, and in any case no other Avodah could be performed until after the Korban Tamid was offered. Therefore the Kohanim not selected would change back into their civilian clothes.
There is a dispute between the Rambam and Ra'avad (Hilchos Kil'ayim 10:32)
whether or not the Kohanim were allowed to remain dressed in the priestly
garments while not performing the Avodah. The Rambam maintains that since
the priestly garments contained Shatnez, it was forbidden for the Kohanim
to wear them while not performing the Avodah. According to his opinion, it is easier to answer your question; since they did not have a Korban to offer at the time, they had to immediately remove their priestly garments, which contained Shatnez.
The Ra'avad, however, argues and asserts that they were permitted to wear the priestly garments all day. Apparently it was nevertheless the practice to remove the garments when not actively performing Avodah, so that they should not risk getting them dirty and disqualifying them from use. (Of course, in the event that the services of the "losing" Kohanim were required during the course of the day in order to offer private sacrifices, they would change back into their priestly garments.)
D. Schloss and the Kollel