Vayikra 10:1,2 record, “Now, Aharon’s sons, Nadav and Avihu, took each-man his pan, and placing coals in them, put smoking-incense on it, and brought near, before the presence of G-d, foreign fire, such as he had not commanded them. And the fire went out from the presence of G-d and consumed them, so they died, before the presence of G-d” (adaptation of Fox’s 1995 translation). It seems clear that Nadav and Avihu died because they did what they were not commanded, but what were they trying to do? Numerous theories have been offered to explain their sin. We will review some of them and then offer our own.
Rashbam (on 10:1) suggests that Nadav and Avihu wanted to offer the incense sacrifice since they thought the regular daily law of offering incense was part of the ritual of the eighth day ceremony and they entered into the mishkan proper to offer the incense even before the fire devoured the sacrifices. Their mistake according to the Rashbam was that on the special day (the eighth day), the fire for the sacrifices was to come from G-d and not from a regular human fire. This approach is problematic since not only does the Rashbam argue that Nadav and Avihu were killed prior to the fire devouring the sacrifices which contradicts the order of the Torah, but also as pointed out by David Tzvi Hoffmann and Shmuel David Luzzatto, they could not have been following the regular law of incense offering since that offering was brought by one priest and not two.
Seforno (on 10:1) proposes that Nadav and Avihu were acting based on the special character of the day. They thought that it was appropriate to offer an incense offering after the fire devoured the special sacrifices of the eighth day, and they were killed when attempting to offer this incense offering on the inner altar in the mishkan proper. According to Seforno, Nadav and Avihu sinned by making this decision to add an incense offering without consulting Moshe. This approach is also difficult because the Torah records that the problem was that their fire was not commanded by G-d and not that the incense offering was not commanded.
Hoffmann (1953, p. 204) writes that Nadav and Avihu wanted to bring the incense offering to show their happiness about the appearance of the fire devouring the sacrifices. He claims that with their pans they brought the incense into the Holy of Holies, like the high priest does on Yom Kippur, though at this point this type of sacrifice had not yet been commanded. Hoffmann explains that while the Torah refers to their foreign fire, this really mean a sacrifice of fire that had not been commanded.
This idea that Nadav and Avihu went into the Holy of Holies is very popular, and it is claimed that 16:1, which refers to Nadav and Avihu’s sin in connection with the command to Aharon to enter the Holy of Holies on Yom Kippur, supports this understanding. Yet, one can understand that in 16:1 the reference to Nadav and Avihu’s sin was to remind Aharon to follow G-d’s command exactly as he was commanded unlike Nadav and Avihu, and does not imply that Nadav and Avihu entered the Holy of Holies. Or, as pointed out by the Ramban (on 16:1), the reference to Nadav and Avihu in 16:1 could be to inform us that the laws regarding Yom Kippur were taught to Moshe immediately after the deaths of Nadav and Avihu. Independent of 16:1, there is no reason to think that Nadav and Avihu would have thought to enter the Holy of Holies on the ceremony of the eighth day since the ceremony of the eighth day was focused on the outer altar which was situated in the courtyard of the mishkan. Thus, it is more likely that they died in the courtyard of the mishkan in front of the mishkan proper (the word kadosh in 10:4).
In order to understand the sin and deaths of Nadav and Avihu we have to connect their actions to 9:24. 9:24 records that a fire went out from the glory of G-d and consumed the sacrifices that were offered as part of the ceremony of the eighth day. After this fire appeared, Nadav and Avihu took their pans, and were walking around the outer altar with their incense offering. It could be that they did this out of joy (Hoffmann) or they thought it was appropriate to have an incense offering at that time (Seforno), but this was not commanded. The gravity of their sin was not the incense offering but the fire used to create the incense offering, the eish zarah, foreign fire, 10:1. The point of the ceremony of the eighth day was that G-d had sanctified the outer altar by sending a fire to devour the sacrifices, and now Nadav and Avihu were walking around the outer altar with a regular human fire. Their fire, which was not commanded by G-d, weakened the boundary between G-d and man since both the human fire and G-d’s fire were spatially co-existent, see our discussion on Shemot 19:12, 21-25, "Establishing boundaries I" https://lobashamayim.blogspot.com/2009/02/shemot-2019-23-yitro-establishing.html. Nadab and Avihu were then killed by another fire that went out from the glory of G-d.