Preheat your oven to 350°F (175°C). Grease a loaf pan and line it with parchment paper. In a large bowl, rub the lemon zest into the coconut sugar with your fingertips until fragrant.
Beat the butter and olive oil into the lemon-sugar mixture until light and fluffy.
Add the eggs one at a time, mixing well after each. Stir in the lemon juice and lightly crushed lavender buds.
In a separate bowl, whisk together the flour, baking powder, and salt.
Gradually add the dry ingredients to the wet mixture, alternating with Greek yogurt.
Begin and end with the flour mixture. Mix until just combined, do not overmix.
Pour the batter into the prepared pan. Bake for 50–55 minutes, or until a toothpick inserted in the center comes out clean.
Let the cake cool in the pan for 10 minutes, then transfer it to a wire rack to cool completely.
In a bowl, whisk together the powdered sugar, lavender extract, lemon zest, and milk or lemon juice until smooth and pourable. Add food coloring if desired.
Drizzle the glaze over the cooled cake. Garnish with dried lavender buds or thin lemon slices if desired.
Slice, serve, and enjoy!