Malbim: They are the wives and children. Only the small children remained as a security that they will return. Mefarshim say that this hints that they wanted to leave, but [Pharaoh] did not allow them. The servitude already began in potential, just it was not carried out until Yosef and that generation died. This is why Yosef said "Ha'Elokim Pakod Yifkod Eschem" (50:25), for they could not leave by themselves.

Ha'amek Davar: "Beis Yosef" are his slaves.