The Gemara (Pesachim 116a) and the commentaries contain many recipes, which are not based merely on taste, but on solid halakhic reasons.
Rabbi Levi says that the charoset is “zekher le-tapuach (in memory of the apple)”, i.e. the apple tree under which the Israelite women hid themselves from the Egyptians to give birth. In order to give the charoset a sharp taste, Rashi says to use apples and wine, and sharp ingredients; Ran says to use sour apples and vinegar.
Rabbi Yochanan maintains that the charoset is zekher le-tit, in memory of the clay which our forefathers made in Egypt. Rashi advises, to thicken it, to put in plenty of ingredients and grind it well, and Ran says to grind many vegetables into it. At the time of the meal, Tosafot says to thin it by pouring in wine or vinegar. Mordekhai, in the name of the Arukh, says it should be made from all types of food – sweet, bitter, and sour – just like clay, which contains all sorts of things.
Leket Yosher, in the name of Terumat ha-Deshem, adds pears. It suggests that if they are mixed in the proper measure with apples and nuts, the resulting material will have the colour of clay. According to other opinions, however, while the charoset should have the texture of clay, it need not be of the same colour.
Shibbolei ha-Leket maintains that, since the charoset is in memory of the clay, we should add a little bit of clay or brick scrapings! Maharam de Lunzano totally disagrees with this, for we must convert sorrow into joy. He says that the suggestion of adding brick scrapings had its origins in a misprint in Rashi and Rashbam (Pesachim 116a). The erroneous text read “ve-cheres”, which means broken bits of pottery, but it is clear that the correct reading should be “ve-charoset”. It is only the latter which is to be ground “in memory of the clay”.
The Talmud Yerushalmi (Pesachim 10:3) says the charset is zekher le-dam, in memory of the blood. Tosafot, quoting the Yerushalmi, accordingly advise the use of wine or vinegar to thin it out at the time of the meal. Rema (Orach Chayim 473:5) rules that it must be made thick like clay, and then we add a little vinegar or red wine, so it will be suitable for dipping.
A Baraita in the name of Rabbi Yochanan says that it is zekher le-teven, in memory of the straw (which was kneaded into the clay). Rashi says to use vegetables; Rashbam, spices; Rif, Rach and Rema all say to use cinnamon or ginger, which resembles straw; Mordekhai (in the name of Rokeach, and quoted by Beit Yosef) suggests calamus, cinnamon, or ginger, which cannot be properly ground to remove the hard strands in them, thereby reminding us of the clay and the straw it contained; Maharil (Sha’ar ha Tziyun 68) writes that one should not grind them, so that they will remain long.
Tosafot, in the name of Teshuvat ha-Geonim, says that charoset is in memory of the fruits to which all Yisrael is compared in Shir haShirim: “Under the apple tree I awakened you” (Shir haShirim 8:5); “The fig tree puts forth her green figs” (2:13); “your temples are like a pomegranate split open” (4:3, 6:7); “I will climb up into the (date) palm tree” (7:9); “I went down into the nut garden” (6:11). Tosafot also adds almonds (shekedim) because the Holy One, blessed be He, shakad, (was eager and diligent) for the End of Days – and to whiten the charoset, since the clay also contained lime.
Peri Chadash, in the name of Kol Bo, adds grapes, in memory of the vines: “and the vines in blossom give forth their fragrance” (Shir haShirim 2:13)
Haggadah Sheleimah (p.63), in the name of Sefer ha-Menuchach, says that it was the custom of Jews from Salonika to add chesnuts and raisins.
Tur (Orach Chayim 473) says that the charoset is in memory of maror, the bitter times, and so sour things should be included in the recipe. He excludes pomegranates, dates, and almonds, because, as Bach explains, only apples, nuts, and figs were mentioned in connection with the Redemption from Egypt. Almonds also are not connected with redemption; to the contrary, they refer to punishment, as in “I see the rod of an almond tree” (Yirmeyahu 1:11)
So we see that there are very many possible ingredients and everyone should follow the custom that their family have adopted and enjoy their individual recipe.