On the bottom of daf 3A of maseches Shabbos, the gemara establishes that a hand (when it is in a different reshus) is different from a body in that yado lo nayech - an outstretched hand (and by extension something resting in an outstretched hand) is not considered at rest whereas a body is. Then on 3B the Gemara poses the question of whether a hand is a carmelis. As part of the question, the Gemara says we know the outstretched hand is not a reshus hayachid nor a reshus harabaim from the cases in our mishna of Baal habayis and Ani (where in the cases that they move their own outstretched hands back into their own reshus they are patur). My question is how can the Gemara know that the outstretched hand is not a reshus harabim or reshus hayachid if we just established that the outstretched hand is not at rest? In other words, maybe the reason the ani and baal habayis are not chayav in the mishna is because the outstreched hand was not at rest? The hand may be a reshus haya
chid or reshus harabaim, but since the outstretched hand (and by extension the object in the outstretched hand) is not at rest, then there is no akirah or hanacha etc? How can the gemara then assume that from the mishna we can derive that the outstretched hand is not reshus hayachid or reshus harabaim?
1. I found your question cited by the Mesivta edition of the Gemara in the name of Divrei Pinchas. He answers that in the Gemara later (5a), Rebbi Abahu states that our Mishnah is referring to a case where the person receiving the object stretched his hand down within 3 Tefachim of the ground. Since the hand is within 3 Tefachim of the ground, this means that it is considered as actually resting on the ground (through the Din of "Lavud"), so in this is case a hand also possesses the status of being at rest. The Divrei Pinchas writes that Abaye in our Gemara, who says that it is obvious that the outstretched hand is not a Reshus ha'Yachid or a Reshus ha'Rabim, agrees with Rebbi Abahu and therefore the hand is considered to be at rest.
2. I would like to suggest a simpler answer to your question. First, it should be noted that Abaye's main purpose is not in fact to state that a Yad is not a Reshus ha'Yachid or a Reshus ha'Rabim, but rather to ask the question of whether or not it is a Karmelis. Abaye's statement about what the hand "is not" is in fact merely an introduction to the Karmelis question. Second, one can say that the reason why the Yad is neither a Reshus ha'Yachid nor a Reshus ha'Rabim is precisely because the hand is not at rest. Rashi writes that it is clear to Abaye that a hand stretched out into a different Reshus does not follow its body entirely. One can explain that the reason is that the hand is not at rest and therefore is not given the same Din as the body, which is stationary.
3. I would also like to point out that you write that the source that teaches that the hand is not a Reshus ha'Yachid or a Reshus ha'Rabim is the Ba'al ha'Bayis and the Ani who are Patur when they move their hands back into their own Reshus. In fact, Rashi states a different reason how we know this:
a. A hand is not a Reshus ha'Rabim because when the Ba'al ha'Bayis takes the bread from the hand of the Ani stretched into the house, he is Patur.
b. It is not a Reshus ha'Yachid because when the Ani takes the loaf from the hand of the Ba'al ha'Bayis, he is Patur.