1)
Were they justified in saying that he will not inherit with them?
1.
Radak (1), Malbim: No. Also a concubine's son inherits him (Yevamos 22b).
2.
If his mother was a harlot, it would be a Safek who is the father, and properly he would not inherit. (PF)
