How could Eliezer discuss returning Yitzchak to there? Yitzchak had never been there!
Riva #1: Since (one of) the Avos 1 was once there, it is called returning.
Riva #2, Roshm Hadar Zekenim (Pasuk 8): It would be a return for Eliezer, so if he would bring Yitzchak there, it is appropriate to call this returning.
Moshav Zekenim: Since Yitzchak's father was once there, it is called returning.