The Me'iri agrees with your take on the matter, and says the Avodas Yachid refers to the avodah done all year round and not Yom Kippur. However Rashi did not learn like that.

Perhaps I can bring a proof to Rashi's explanation from the fact that the Gemara says, "After finishing the public service". If this was referring to an ordinary Kohen doing an "Avodas Yachid", he would not necessarily have done any public service before. A more likely proof is from the context of the Gemara. According to Rashi's Girsa (i.e. the Girsa in our Gemaras), the preceding Gemara is talking about the Kohen Gadol on Yom Kipur, and the subsequent Gemaras are also referring to the Kohen Gadol making it at least a strong possibility that this Gemara is also talking about the Kohen Gadol, and Yom Kipur.

The Ritva, however, brings a strong proof that this Gemara isn't referring to the private Karbonos of the rest of the year. He says, quite simply, private Karbonos required the real priestly garments! (That is, the Gemara distinguished between Avodas Yachid and Avodas Tzibur, as far as wearing home-made garments, presumably because there is a possibility that the person who made the garments will not give them to the Tzibur wholeheartedly. However, there is no logical source to differentiate between personal and public Korbanos as far as wearing Bigdei Kehunah. The Avodas Yachid of our Gemara must be one for which Bigdei Kehunah and not absolutely necessary.) This, he says, is why Rashi explains the Avodas Yachid to mean the removal of the fire pan and spoon from the Kodesh Ha'Kodashim.

(The Gemara on 25a that says that clothes the Kohen's mother made for him were permitted to be used for the "Avodas Yachid," although it is referring to ordinary Kohanim. However, it does not mean that the ordinary Kohanim performed Avodas Yachid. It just is trying to bring an example of Kehunah-like clothes that were made by private individuals -- in this case, by the Kohen Gadol's mother.)

D. Schloss and M. Kornfeld