How could he say that Binyamin will die through leaving his father? He left, and did not die!
1.
Ha'amek Davar: Only Azivah (leaving absolutely) will kill him.
1
Yosef did not demand that Binyamin be by him permanently, rather, only when the brothers are in front of Yosef.