Begin by preheating your oven to 350 degrees F. Line an 8-inch square baking pan with parchment paper, leaving an overhang on two of the sides. Spray the sides and bottom with cooking spray.
In a bowl, stir together the butter, sugar, and graham cracker crumbs. Press the mixture into the bottom of the prepared baking pan. Bake until the edges are just starting to brown, about 5 to 8 minutes. Transfer pan to a wire rack and let cool for at least 30 minutes.
Meanwhile, in a large saucepan over medium heat, stir together the condensed milk, white chocolate chips, and chopped white chocolate until the chocolate is melted. Constantly stir the mixture to avoid burning. Remove the pan from the heat and then stir in the lime zest, lime juice, and vanilla extract. Pour the mixture over the cooled graham cracker crust. Chill for 4 hours or overnight.
Lift the fudge from the pan using the parchment paper. Then cut the fudge into bite-size pieces. Store in an airtight container for up to a week.