Preheat your oven to 350°F (175°C). Grease a 9x5 inch loaf pan and line it with parchment paper.
In a medium bowl, whisk together the gluten free baking flour, almond flour, ginger, cinnamon, cloves, nutmeg, baking powder, baking soda, and salt. Set aside.
In a large bowl, beat the softened butter and dark brown sugar with an electric mixer until fluffy, about 3 minutes. Add the molasses and beat until incorporated. Beat in the eggs one at a time, then mix in the vanilla extract and buttermilk until smooth.
Fold the dry ingredients into the wet ingredients using a spatula until just combined. The batter will be thick and deeply aromatic with warm spices.
Pour the batter into the prepared pan and smooth the top. Bake for 50 to 60 minutes, until the top is domed and set and a toothpick inserted in the center comes out with a few moist crumbs.
Let the bread cool in the pan for 15 minutes, then transfer to a wire rack to cool completely.
While the bread cools, beat together the cream cheese, powdered sugar, milk, and vanilla extract until completely smooth and thick but drizzleable. Drizzle generously over the cooled bread, letting it drip down the sides. Let the glaze set for 10 minutes before slicing.