This is the question Rashbam is dealing with. The only time we can say Peh She'asar is where the they are saying in the "Hitir" that their signatures are false. However, if they are admitting that their signatures are true they cannot testify against the Shtar.

With regard to Shtar Amanah there is an additional reason: Since they are not allowed to sign on an Amanah in the first place they are not believed with the "Hitir" since "Ein Adam Mesim Atzmo Rasha."

D. Zupnik

Here is a copy of some Insights the Kollel wrote on that Rashbam, that might be related to your question:

====================================================

Bava Basra 49

1) BELIEVING WITNESSES WHO WANT TO INVALIDATE A "SHTAR" THAT THEY SIGNED

QUESTION: Rav Nachman rules that when two witnesses who signed a Shtar testify later that the Shtar was a Shtar Amanah (that is, there was not really a loan), they are not believed. Similarly, when two witnesses who signed a Shtar testify later that, before they signed the Shtar, the seller declared a Moda'a in their presence, saying that the transaction was done under coercion, they are not believed. Mar bar Rav Ashi argues and says that when they testify that there was a Moda'a, they are believed.

The RASHBAM (DH Ein Ne'emanim) writes that when the Gemara says that the witnesses are not believed to say that the Shtar they signed is a Shtar Amanah, this applies not only when the Shtar is Mekuyam, but it applies even when the Shtar is not yet Mekuyam. The witnesses are still not believed to say that the Shtar was an Amanah, because of the principle of "Keivan she'Higid Eino Chozer u'Magid" -- once they have given testimony, they cannot change that testimony. Their signatures on the Shtar is considered to be their original testimony which cannot be altered afterwards. The Rashbam proves this from the Gemara in Kesuvos (18b). How, though, can the Rashbam prove this from the Gemara there? The Gemara there is discussing a Shtar that is Mekuyam . If the Shtar is not Mekuyam, then the witnesses are believed to say that the Shtar is not valid, because the validity of the Shtar itself is based on their testimony and thus they are believed because of the principle of "Peh she'Asar Hu ha'Peh she'Hitir," as the Rashbam himself mentions later! (RASHASH)

ANSWERS:

(a) The RASHASH explains that the Rashbam is not proving from the Gemara that "Keivan she'Higid" applies when the Shtar is not Mekuyam. Rather, he is proving only that "Keivan she'Higid" applies to testimony that is written in a Shtar.

Why, though, does the Rashbam write that witnesses cannot say that the Shtar is an Amanah even if the Shtar is not Mekuyam, if the Gemara in Kesuvos says that they have a "Peh she'Asur?" The answer is that the Gemara there only gives credibility to the witnesses because of "Peh she'Asar" when they are not diametrically contradicting what is written in the Shtar. When the witnesses say that the Shtar was an Amanah, they are contradicting what is written in the Shtar, and therefore they are not believed (see TOSFOS DH Amar Rav Nachman).

(b) However, the Rashbam does not seem to be saying this. In a few places, his words imply that when the Shtar is not Mekuyam, the witnesses are believed through "Peh she'Asur," and their second testimony is not invalid because of "Keivan she'Higid" (see Rashbam DH u'Mar bar Rav Ashi, and DH Moda'a Hayah Devareinu, where he writes that there is no "Peh she'Asur" because we know that the Shtar is valid because the lender and borrower both agree that it is valid). In addition, Tosfos suggests this logic only to explain why "Peh she'Asur" does not apply according to his own view that Rav Nachman is discussing a Shtar that is not Mekuyam.

Therefore, it seems that when the Rashbam writes that the reason the witnesses are not believed is because of "Keivan she'Higid," he is referring back to the case of a Shtar that is Mekuyam. Only later does he give the reason for why the witnesses are not believed when the Shtar is not Mekuyam, when he writes that the witnesses cannot give testimony that will incriminate themselves.