The Rama (Cracow, 1520-1572) writes (on the Shulchan Arukh, Orah Chayyim, 583:2) that on Rosh Hashanah people go to the rivers and recite the verse from Micah (7:19) "G-d should cast into the depths of the seas all their sins." The word for casting in Hebrew is tashlikh and this has become the name of this custom. However, the Rama only briefly mentioned the custom and he left out (probably purposely) the main element in the custom, which is the throwing of food to the fish.
Jacob Lauterbach (1873-1942) has a very lengthy review of the tashlikh custom. He quotes (1936, reprinted in 1973, p.389) from a convert, John Pfefferkorn (1469-1521) who wrote that "the people would come to near the water, shake their garments, (in which they carried their food) and, throwing the food into the water, they would cry out to the fish, 'We throw our sins to you." Pfefferkorn also did woodcuts of the ceremony and he shows the same idea of people throwing food into the war to the fishes. Lauterbach notes that another convert, Anthonius Margaritha gave a similar description of tashlikh in 1530.
The custom is first mentioned by the Maharil (R. Yaakov ben Moshe Levi Moelin, Germany, 1365-1427, Laws of Rosh Hashanah) and he writes that after the meal on Rosh Hashanah, people go to a river and recite the verse of Micah 7:19. Lauterbach (p.385) suggests that the timing of doing tashlikh after the meal was because people took the leftover food from the meal to throw to the fish. However, the Maharil writes that one should not throw food to the fish. From this statement, we see that the custom was in fact to throw food to the fish. (This is how I remember the custom when growing up in West Hempstead, Long Island in the early 1970s.)
It is not clear why the Maharil did not accept the custom of throwing food to the fish. The Maharil writes that throwing food to the fish violates the laws of carrying on the festivals and of feeding animals that are not yours. However, there is no prohibition of carrying on the festival if the carrying is for the festival and here the carrying of food is to fulfill the custom of the festival, which makes it permissible. With regard to feeding animals, the Shulchan Arukh (497:2) rules that on yom tov cannot feed fish, birds and animals that are not your own since you might come to catch them. The Mishnah Berurah (497:5) comments that it is only forbidden to feed the animals by placing the food right in front of them but if one puts the food a little away from them and they come on their own to get the food it is permitted. Thus, by tashlikh on yom tov there is no prohibition of feeding the fish since the person is not intending to catch the fish and the fish come up from the water to get the food (usually bread and crumbs).
It is possible that the Maharil, who lived before the Shulchan Arukh, did not accept the Shulchan Arukh's ruling on feeding animals, but more likely his two reasons were a cover, and he had another reason which he did not want to publicize. Daniel Sperber (1995, p.121) makes the interesting suggestion that the fear was that Christians would seize upon the throwing of food into the river as a pretense to claim that the Jews were poisoning the water. Another possibility is that the practice of throwing food to the fish was copying a Christian practice, as Lauterbach (p.429, also noted by Sperber, p.119) writes that Petrarch recorded a ceremony exactly like tashlikh done by Christians in Cologne, Germany in the 14th century. (Who copied who?) A third possibility is that the Maharil was trying to minimize the custom since the going to the river and hanging around to feed the fish led to the intermingling of men and women, see Arukh Hashulchan (583:4), who as much later time period, end of the 19th century, was concerned about this issue.
Due to the rabbinic protestations against the custom of throwing food to the fish and in some places the lack of water with fish, the custom of tashlikh has evolved into various forms. One practice is to continue the initial custom of reciting the verses and throwing food to the fish. I believe this practice is still prevalent in the Diaspora. Two, amongst Sefardim and many in Israel, one just looks at some body of water and recites the verses, without throwing any food into the water. This body of water can be water from a faucet, a hose, a fountain or a water tower, where one does not see the water but knows that there is water inside. Within this variation, some people make an effort to go to a body of water that have fish, but they do not throw food to the fish. A third variation is that one goes to the water, recites the verses, and empties one's pockets. This emptying of the pockets is clearly the idea of taking the crumbs out of the pockets to feed the fish, but today most people's pockets are clean, so they are just shaking out empty pockets. A fourth variation, which I have never seen, is quoted by Lauterbach (pp.399, 411) that people in Tangiers in 17th century and people from Kurdistan in the 19th century, actually jumped into the war themselves! Finally, it is quoted (Ma'aseh Rav) that the Gra (1720-1797) did not do tashlikh at all, and the Arukh Hashulchan (583:4) writes that many people do not do this custom.
What is the reason for the custom of tashlikh? The Maharil (quoted in Mishnah Berurah 583:8) writes that it is to recall a miracle that G-d did for Avraham on the way to the akedah. According to the Midrash, which was written before the custom of tashlikh existed, the devil tried to block Avraham from going to do the akedah by becoming a raging river, but G-d did a miracle to dry up the river (the devil). This rationale for tashlikh is very strange. Why should we want to go to a river since that represents the devil? With this logic one should do tashlikh by a dry place. However, we need to remember that the Maharil was not trying to explain the real custom but was trying to find a reason for his desired change in the custom that people would just go to the river and not throw food into the river.
The Rama in his commentary on the Tur, the Darkei Moshe, which he wrote before his commentary on the Shulchan Arukh, gives another reason for the custom. In the Darkei Moshe (583), he quotes from R. Isaac Tyrnau (contemporary of the Maharil, Sefer ha-Minhagim) that the point was to see fish, which somehow is a sign that the evil eye will not look upon us (do not really understand) since it does not look upon the fish (not sure why not). The Rama in a different book, Torat ha-Olah 3:56 (quoted in Sperber, p.120) gives another reason that by going to the river this reminds one of G-d's majesty. This rationale is obviously problematic for one who fulfills the custom of tashlikh by turning on a faucet. For other reasons and more sources on this custom, see Rabbi Ari Zivotosky's fascinating review of the custom in Jewish Action, Fall 5768/ 2007, pp. 62-65, http://www.ou.org/torah/
All these reasons are difficult since they are not explaining the real custom of throwing food to the fish, but the real custom has a simple explanation. On Yom Kippur, a major part of the ritual was for the high priest to transfer the sins of the people to a goat which was sent into the desert, Vayikra 16:21,22. This ceremony ended with the destruction of the Bet ha-Mikdash, but because this was such an important part of the ritual of the day, people wanted to continue it in some other forms and tashlikh is one of these attempts. Thus, the food, which symbolizes the sins, is sent to the fish who will take the sins far away. The Magen Avraham (1635-1682, 583:5) notes that it is best to do tashlikh outside the city, which is the idea of sending the sins far away. This understanding of the custom is not novel. It is exactly the meaning of the verse that is recited by tashlikh, Micah 7:19, to send the sins to the depths of the sea. Lauterbach (pp.397-399) quotes even two non-Jews from the 17th century who write that the custom was to carry the sins off "just like the scapegoat of old." With this understanding one should not do tashlikh by feeding fish in a home aquarium since then the sins stay at home.
Yet, the sending away of the sins is on Yom Kippur, why is this ceremony done on Rosh Hashanah? The answer is that there exists another custom, kapparot, which has the same symbolism to send away the sins, see our post Kapparot, http://www.lobashamayim.blogspot.co.il/2011/10/kapparot. Yet, kapparot is the older custom and is done on erev Yom Kippur, so why was another custom added which has the same meaning? To answer this question, we need to determine when the custom of tashlikh began.
The Maharil was the first person to quote the custom of tashlikh, which suggests that the custom began in his lifetime or shortly beforehand. Lauterbach (p.429) notes that people have made this argument but he thinks they are deluded. Yet, I think the argument is correct since the Tur (1269-1343?) who lived prior to the Maharil, quotes all Ashkenazi customs in his book, the Tur, but he does not refer to the custom of tashlikh. This strongly implies that he was not aware of the custom since it did not exist.
Why did the custom begin around the 14th century? I believe that there are two possible complementary answers. One, Rosh Hashanah is considered the day of judgment, and hence there developed the idea that people need to remove their sins by Rosh Hashanah and not wait until Yom Kippur. Two, the 14th century was a very difficult period for the Jews. The Black Plague killed off 50-60% of the population of Western Europe, and in the ensuing anti-Semitism, more Jews were killed since they were blamed for causing the plague. Thus, life began to be viewed as being very tenuous, even more so than before, and this created a need to do everything possible to be judged favorably. Tashlikh was an answer to this need since this was a symbolic act to hopefully increase one's chances of living. Note the Maharil was also the first person to quote the idea that one has to do kapparot for a fetus (see Darkei Moshe 505) which again is the idea of doing everything to ensure that G-d would judge the unborn child (who could not even have sinned) favorably.
Tashlikh became a crucial part of Rosh Hashanah, and then there developed the problem of what to do when the first day of Rosh Hashanah is on Shabbat because then there is truly a problem of carrying the food to the river. There would be no problem of carrying if the river was within the eruv (theoretical door frames that allow one to carry), but not all places had eruvs and as mentioned above tashlikh was usually done outside the city, which would be outside the eruv. This problem must have arisen almost immediately with the development of the custom, but Lauterbach (p.420) writes that R. Jehiel M. Epstein, the author of the Kitzur Shnei Luchot ha-Brit (end of 17th century) was the first person to discuss this problem, some 200-300 years after the custom started. R. Jehiel M. Epstein argued that when the first day of Rosh Hashanah was on Shabbat, tashlikh should be pushed off to Sunday, the second day of Rosh Hashanah. Lauterbach notes that R. Jehiel M. Epstein also objected to the "popular" idea that by tashlikh one throws away one's sins, so then for him it was not difficult to move the day. However, for those people who accepted the symbolism of tashlikh, it was very important to have tashlikh on the first day.
In 1725, the first day of Rosh Hashanah was on Shabbat and R. Jacob Riescher (Shvut Yaakov 3:42) argued that one should do tashlikh on the first day and that the Maharil allowed tashlikh on Shabbat. Yet, what about the prohibition of carrying on Shabbat? One could claim that the proponents of doing tashlikh on Shabbat were only maintaining that one should recite verses by the river and these verses could be memorized. Yet, again the practice was to throw food to the fish, so how could one carry the food to the river? Maybe the people usually threw food, but when the first day of Rosh Hashanah was on Shabbat then tashlikh was done without throwing food. Another possibility is that in his discussion of tashlikh, the Maharil writes that one should not take food from a non-Jewish person who was by the river. Lauterbach (p.386) notes that the non-Jew's presence by the river was surely no accident, and this suggests that on Shabbat the people would have non-Jews bring the food for them to throw into the water. If this is true, then it is quite amazing that we do not blow the shofar on Shabbat due to fear of carrying, but people would continue to do tashlikh on Shabbat. (With regard to throwing the food into the water, this would be from a carmelit (the edge of the river, see Mishnah Berurah, 301:176) to a carmelit (the river) less than four amot, which is permitted, see Shulchan Arukh 346:2. Even if the water would take the food further away, this would be coah (power) of the carmelit which is permitted see Shulchan Arukh, 355:1)
The Mishnah Berurah (583:7, end of 19th century, beginning of the 20th century) writes that he saw that in some places tashlikh was done on the second day of Rosh Hashanah when the first day falls on Shabbat because of the problem of carrying the machzor, but he does not refer to problem of bringing food to the river. The implication of his words (a few places) is that still in his time most people did tashlikh on the first day of Rosh Hashanah even when it was on Shabbat, see also Shaarei Tshuvah 583:10 (R. Chaim Mordechai Margoliot, 19th century). My impression is that today most people do tashlikh on the second day of Rosh Hashanah when the first day of Rosh Hashanah is on Shabbat.
In conclusion, the custom of tashlikh developed around the 14th century, and the initial custom was to throw food to the fish. This was thought to symbolically remove one's sins in order that a person would be judged favorably on Rosh Hashanah. This became such an intrinsic part of the day that even when the first day of Rosh Hashanah was on Shabbat, initially many people still did tashlikh, possibly by having a non-Jewish person bring them the food to throw.