There are three approaches to understanding Moshe's order to the Levites. One, Moshe set up a court system to judge the people, and apparently the Levites were both the officials of the court and the executers of the court's judgments. (See Yoma 66b, also the Midrash Meor ha-Afalah, who is quoted by Menachem Kasher, Torah Shelemah 32:114, and Kasher's comments 114. In addition, see Midrash ha-Gadol, who is also quoted by Kasher, 32:225, Kasher's comments on 225, and Arussi 2000.) With this approach, there was no indiscriminate killing. However, as noted by the Ramban (on 32:27), this approach is problematic since there were 600,000 men and if there was a trial for each man (also the women?) then this would have taken years, while the impression of the Torah is that the actions of the Levites transpired immediately.
A second approach is from the Ramban who writes that this incident was a unique case to sanctify G-d's name, and did not follow normal judicial procedures, as for example there was no one around who could have warned each sinner. However, he also maintains that the Levites did not kill indiscriminately, but only those people they personally knew who had sinned.
A third approach is from the Ibn Ezra (short commentary on 32:27) who mentions the idea that the Levites only killed those people they knew had sinned, but he adds that they also killed those people who had a sign on their bodies that they had worshipped the golden calf. Ibn Ezra elaborates on this idea in his long commentary on 32:20. Ibn Ezra explains that when Moshe came down from the mountain he made the people drink the ashes of the golden calf with water, 32:30, and this caused the bodies of the people who had sinned with the golden calf to swell or for there to be marks on their faces similar to the water of the suspected adulteress (Bemidbar 5:27, see Avodah Zara 44a). The disfigured bodies or marks indicated to the Levites which people deserved to die.
Ibn Ezra's explanation is logical since 32:20 records that Moshe forced or caused the people to drink the ashes, and there is no record of any consequence to their drinking of the ashes. Why would Moshe cause the people to drink the water for no reason? However, once we know the case of the suspected adulteress, who would drink ashes which would cause marks on her body, then we can infer that Moshe forced the people to drink in order for there to be marks on their body, and the reason why the people had to have marks on their bodies was in order that the Levites could identify which people to kill. Accordingly, G-d did a miracle to identify the people deserving of punishment and this obviated the need to have courts. (Rashi on 32:20 based on the Talmud implies that the drinking per se caused the people to die, which is difficult since presumably more than 3,000 people drank the water, 32:20.)
While the Ibn Ezra's explanation is logical, how did Moshe get everybody to drink the water? Ibn Ezra writes that he commanded the people to drink, but would the people listen to such an order? Could a person be forced to drink water? The answer is from Moshe's' recollection of the sin of the golden calf in Devarim 9:21, where Moshe explained that he poured the water into the stream that flowed from Mount Sinai.
What seems to have happened is that Moshe went to the source of the water on Mount Sinai, and poured the ashes into the stream but the people did not see this. The people then naturally drank from the water as it flowed into the camp, and when they did this, the bodies of those people who were guilty were marked. Thus, Moshe caused or forced the people to drink the ashes, see Bekhor Shor on 32:20. This would be the same idea as in modern times if the government puts fluoride in the main water pumps of an area, and then everybody drinks the fluoride knowingly or unknowingly.
With this approach, we can understand the order of the events when Moshe came down from Mount Sinai. First, Moshe broke the tables, 32:19. Then, Moshe pulverized the golden calf, 32:20, and poured the ashes into the stream, Devarim 9:21. He did this away from the people on the Mount Sinai, and it took some time. He probably finished sometime in the afternoon, and he had hoped that the people would have quieted down after he had broken the tablets. Only after he had destroyed the idol did he confront Aharon as to what happened, 32:21-24. This conversation would have occurred in the late afternoon. At this point, the people were still out of control, 32:25, and when they did not respond to him after he asked the people, "who is with G-d, come to me," 32:26, he gave his order to the Levites to kill the people.
By this time, the late afternoon, there had been some time for some of the people to have drank the water and for the marks on their bodies to develop. It seems that not all the people who had rejected G-d were killed, but rather the 3,000 people killed were enough to restore order in the camp, and Moshe went to pray for the others, 32:30. Or, maybe, these 3,000 people had been particularly egregious sinners either in not listening to Moshe in 32:26 or by the golden calf. Note if the people had responded positively to Moshe's question, "Who is with G-d" in 32:26, then they would not have been killed. Either there would have been no marks from drinking the water, as G-d would not have caused the miracle to occur, or maybe the people would have remained with marks throughout their lives, but they would not have been killed.
Ramban (on Devarim 9:21) asks why does Shemot 32:20 record that Moshe forced the people to drink the ashes, while Devarim 9:21 just records that Moshe put the ashes in the stream. He suggests that in Devarim, Moshe was showing respect for the people since he did not want to recall that he had treated them as a suspected adulteress. Our approach here offers a slightly different answer.
In the description of the sin of the golden calf in Shemot, the Torah had to record that Moshe forced or caused the people to drink the ashes since otherwise one might have thought that Moshe just poured ashes into the water to destroy the idol (see Bekhor Shor and Hizkuni on Shemot 32:20). This language of causation implies that there was a reason for the drinking of the ashes, and this reason was for the Levites to be able to identify the sinners. On the other hand, in Devarim, Moshe did not explicitly refer to the Levites' actions (see Devarim 10:8), and consequently he did not have to state that he forced the people to drink the ashes. Instead, in Devarim, Moshe was explaining how he got the people to drink the ashes, and hence he just said that he put the ashes into the stream. Thus, the two verses, Shemot 32:20 and Devarim 9:21, complement each other.
Finally, if G-d performed a miracle that the ashes enabled the Levites to identify the sinners, why did G-d not simply strike those sinners or have the people die from the ashes without the Levites killing the people? One answer is that this is the usual method of the Torah to minimize miracles. In this case, the judicial system was impossible, so G-d performed a miracle, but just to identify the sinners. A second answer is that the steadfastness of the Levites would later lead the people to accept them as replacements of the firstborn sons, Bemidbar 3:41,45, and Devarim 10:8.
Arusi, Ratzon, 2002, The trial of those who worshipped the golden calf, A divinely given Torah in our day and age, Vol. II, edited by Aryeh A. Frimer and Shlomo H. Pick, Ramat Gan: Bar Ilan University Press, pp. 245-249.
Kasher, Menachem (1895-1983), 1992 (first volume appeared in 1927), Torah Shelemah: A talmudic-midrashic encyclopedia of the five books of Moses, Jerusalem: Beth Torah Shelemah, the Torah Shelemah Institute.