What exactly, did Nadav and Avihu do wrong?
Rashi #1 (on Pasuk 2, citing R. Eliezer, from Eruvin, 63a): They ruled a Halachah in the presence of their Rebbe Moshe. 1
Rashi #2 (2 citing R. Yishmael from Eruvin, 63a): They drank wine 2 before entering the Mishkan.
Ramban (2, according to Kabalah): They brought the Ketores to the Midas ha'Din, and not directly to Hashem. 3
Seforno: They brought private Ketores in honor of the advent of the Shechinah 4 - just as the Ketores Tzibur was brought daily after the Tamid shel Shachar (via which the Shechinah dwelled in Yisrael). 5
Rashbam, Rosh (2), Da'as Zekenim: They brought human fire before fire from Heaven descended for the first time. 6 Once Heavenly fire descended, it was a Mitzvah to bring human fire.
Moshav Zekenim: They should have brought the Ketores on the Mizbe'ach, and they brought it in incense pans.
Refer to 1:7:151:2 and the note there.
Moshav Zekenim (16:1, citing Vayikra Rabah 20:10): When Moshe and Aharon walked in front of them, thexy said 'when these two elders die, we will have authority over the Tzibur 7 .' It says "b'Karvasam", for they said 'let the day of their death come close.'
Moshav Zekenim (16:1, citing Vayikra Rabah 20:8): It was for four matters - Kereivah (they entered the Kodesh ha'Kodoshim), they offered a Korban that they were not commanded, they brought strange fire, and they did not take counsel from each other.
Moshav Zekenim (16:1, citing Vayikra Rabah 20:9): It was for four matters about which Misah is written - they entered after drinking wine, and without washing the hands and feet, and Mechusar Begadim; they lacked the Me'il - "v'Nishma Kolo
That although fire would come down from Heaven, it was a Mitzvah to add human fire (Rashi). That one should bring private Ketores in honor of the Shechinah (even assuming that it was intrinsically correct to do so Seforno).
Which explains why the Torah follows this Parshah with the prohibition against drinking wine (See Pasuk 8, 9).
See Rav Chavel's commentary, citing R. Bachye. Refer also to Shemos 30:1:2:1. See also Ba'al ha'Turim.
Rosh, Da'as Zekenim: This diminishes Hashem's honor, for people will say that human fire consumed the Korban. However, this is difficult, for the previous Pasuk says "va'Tetzei Esh mi'Lifnei Hashem"!
Michtav me'Eliyahu (2, p.249, citing Chafetz Chaim on Sifra): They said so to investigate themselves to be ready for this great responsibility. Their sin was callousness - they should have said 'if, Heaven forbid they will die
What does it mean "Asher Lo Tzivah Osam"?
Da'as Zekenim: They were already forbidden "Lo Sa'alu Alav Esh Zarah" (Shemos 30:9), and Hashem did not command them [to do so for a Hora'as Sha'ah].
Hadar Zekenim, Moshav Zekenim: We invert the verse to read 'Asher Tzivah Osam Lo Sakrivu'. 1
Hadar Zekenim: We similarly invert "va'Yelech va'Ya'avod Elohim Acherim
Why did Tzadikim like Nadav and Avihu do this sin?
Moshav Zekenim citing R. Yehudah ha'Chasid: They longed to see the radiance of the Shechinah; they said that through burning [Ketores] in the Heichal, we will see it overtly.
Vayikra Rabah (20:9) says that they were Mechusar Begadim; they lacked the Me'il. Only a Kohen Gadol wears the Me'il!
Teshuvas ha'Rosh (13:21) #1: It says in Zevachim (102a) that they were considered Seganim. The Segan serves with eight Begadim (Horiyos 12b). 1
Teshuvas ha'Rosh (13:21) #2: They considered themselves like Kohanim Gedolim. According to their mistake, they were lacking the Me'il.
Yedei Moshe (on Vayikra Rabah): The Ra'avad says that a Kohen is like a Kohen Gadol on the day he is anointed. 2
Yedei Moshe (on Vayikra Rabah, citing R. Naftoli Katz): Had they worn the Me'il, the angels would have departed and not killed them.
Rosh: However, some say that this is only if the Segan once serve in place of the Kohen Gadol. Also, only Nadav was the Segan (prepared to serve in place of Aharon, if needed. Rashi (102a) says that Elazar was the Segan (after they died. Perhaps he holds that both Nadav and Avihu were Seganim in their lifetimes. - PF)
This implies that every Kohen must serve in the eight garments on the day he is anointed. If so, why did Moshe dress Aharon's sons in the four garments (8:13), if they could not serve in them? (PF)
QUESTIONS ON RASHI
Rashi writes that they were liable for drinking wine before entering the Mishkan. They were not commanded until after this!
Moshav Zekenim citing the Rosh, Riva (9:24): They were burned primarily due to Chet ha'Egel 1 (refer to 10:12:1:1*). R. Chaim Paltiel (2) - or, they were liable from Matan Torah - "v'El Atzilei Bnei Yisrael Lo Shalach Yado" (Shemos 24:11).
Ramban (9), Moshav Zekenim: Drinking wine caused them to have wrong thoughts (Refer to 10:1:1:3).
Michtav me'Eliyahu (2, p.247): Even though they were not commanded yet, had they had the proper fear, they would not enter after drinking even a Revi'is.
Rashi writes that they were liable for ruling in front of their Rebbi, or drinking wine before entering the Mishkan. It says ("b'Hakrivam Esh Zarah" (Bamidbar 3:4)!
Moshav Zekenim #1: They should have brought coals from the outer Mizbe'ach, like it says in Tamid (5:5), and they brought fire from outside.
Moshav Zekenim #2: They should have brought coals - "v'Lakach
Eliyahu said so not for the honor of Hashem's fire, rather, to prove that the idolatry is impotent! (PF)