What exactly, does "Vayisnaklu oso Lahamiso" mean?
Ramban (citing Bereishis Rabah 84:14): It refers to the previous phrase ("uv'Terem Yikrav Aleihem"). They planned to kill Yosef (before he even arrived), by setting dogs on him 1 - in order to avoid having to murder him directly). 2
Rashi: "Oso" is like Ito, Imo, i.e. Eilav (about him). 5
Ha'Emek Davar: They judged that he is Misnakel (planning to kill) them, and therefore they are justified to kill him.
Nezer ha'Kodesh (on the Midrash): We learn (Pesachim 118a) from "la'Kelev Tashlichun Oso" (Shemos 22:30) that one who speaks Lashon ha'Ra (as Yosef had done) deserves to be thrown to the dogs.
And it was only when that plan of action failed that they discussed killing him directly when he arrived.
Ha'Emek Davar: If so, it should not say "Lahamiso"!
Seforno: So that he alone of all Ya'akov's sons would remain blessed. And the Torah says that if someone comes to kill you, beat him to the draw and kill him first' (Sanhedrin, 72a). Otherwise, following all the harm that they caused him and the lack of pity that they displayed towards him when he pleaded with them, it is inconceivable that they would all have been considered complete Tzadikim and that their names would have appeared on the Eifod.
Divrei Eliyahu, Kol Eliyahu: "Va'Yisnaklu" is plotting against, like "Asher Niklu Lachem Al D'var Pe'or." It is not appropriate to say "Oso"! (This pushed Seforno to explain that they assumed that Yosef plotted against them - refer to 37:18:1:2 - PF). Therefore, Rashi explains that Oso is like Ito, and we find that Ito and Imo can mean 'about him' or 'to him', e.g. "Asher Hufkad Ito" (Vayikra 5:23), "Ledaber Ito" (Shmos 34:35), "Lo Na'aseh Imo Davar" (Esther 6:3).
How could the Shevatim, who were great Tzadikim, plan to kill Yosef?
Moshav Zekenim: He was Chayav Misah by making himself idolatry, through saying "the moon and stars are bowing down to me."
R. Chaim Paltiel: Yosef said that they transgress Arayos and Ever Min ha'Chai, for which a Ben Noach is Chayav Misah, so he is like a Rodef, whom one kills to save the victim. Malbim - they judged that he is like a Rodef trying to eradicate them from this world and the world to come.
Shlah (Vayeishev 40): They judged that he had rebelled against Malchus Beis David (from Yehudah).
Refer to 37:18:1:2*.
Bereishis Rabah (84:14): They saw that later, he (i.e. Yarav'am, who descends from him) would lead people to serve idolatry. 1
Kli Yakar (26): A talebearer causes murder. Lashon ha'Ra kills three (the speaker, listener and subject) - and if someone seeks to kill you, kill him first!
Or ha'Chayim: Yosef falsely told Yakov that they transgressed Ever Min ha'Chai or Arayos. A Ben Noach is killed for these, even through one witness, even a relative, even without warning. They judged him to be an Ed Zomem. 2
Afterwards, Yosef told them - just like you judged your brother based on the future, so I judge Binyamin (Aderes Eliyahu of the Ben Ish Chai, Vayigash) or you (to be Meraglim - B'ris Torah Miketz DH v'Yuvan). Refer to 37:17:1:3.
Ed Zomem is only one who was not where he claimed to see the testimony, and he is killed only if the defendant was sentenced to die through his testimony! Perhaps these conditions are only for Beis Din to kill him, but any witness who seeks to kill through false testimony is Chayav Misah bi'Ydei Shamayim. (PF)
Why does it say that they saw him from afar "and before he came close..."?
Ohr ha'Chayim #1: This teaches that he was very far away.
Ohr ha'Chayim #2: This teaches that "from afar" means far from brotherhood, and "before he came close" is another matter.
Ha'Emek Davar (19): Before he came close, they looked for a way to kill him without overtly spilling blood. They did not find a way until he came close.