“You will cast all your sins into the depths of the sea.” (Micah 7:19)
On the afternoon of the first day of Rosh Hashanah it is tradition to walk to a river or spring (preferably one that has fish in it) and recite special prayers called penitential prayers. The Tashlich prayer comes from the Book of Micha (7:18-20) in the TaNaCH. After you say the prayers you throw the bread crumbs into the water.
Some people keep a piece of the Afikoman from Passover for this purpose which is a nice way of linking the holidays together.
Tossing the bread crumbs on the water is symbolic of casting away our sins and starting a new year with a clean slate.
If the first day of Rosh Hashanah is on Shabbat then Tashlich is traditionally done on the second day.