The Ritva understands that Rashi means that Rebbi Tzadok held that Chazal decreed that an Am ha'Aretz causes impurity even if he touches the back of an earthenware vessel. Since the olives were wet, they would also become impure. (The Rishonim, including the Ritva, have difficulty with Rashi.)

All the best,

Yaakov