Cut or tear day-old French bread into 1-inch cubes (about 10 cups). If using fresh bread, dry cubes on a baking sheet at 300°F (150°C) for 10-15 minutes until dry and slightly crisp.
If using fresh peaches, peel by scoring an X on the bottom, blanching in boiling water for 30-60 seconds, and transferring to an ice bath. Remove pits and dice into 3/4-inch pieces (about 3 cups). Drain canned or thawed frozen peaches well and pat dry.
In a large bowl, whisk eggs, light brown sugar, and granulated sugar until pale (1-2 minutes). Add flour, cinnamon, nutmeg, and salt, whisking until smooth. Pour in milk, heavy cream, melted butter, and vanilla, whisking until fully incorporated.
Grease a 9x13-inch baking dish with butter. Add cubed bread in an even layer, then scatter diced peaches over and among the bread. Slowly pour custard evenly over the top, gently pressing bread down to absorb. Let rest at room temperature for 20-30 minutes or refrigerate for 1-2 hours (or overnight).
Preheat oven to 350°F (175°C). Cover dish loosely with foil and bake for 30 minutes. Remove foil and continue baking uncovered for 25-35 minutes until golden brown and a knife inserted in the center comes out clean. Internal temperature should reach about 160°F (71°C).
Let pudding rest for 10-15 minutes before serving. Meanwhile, in a small saucepan over medium heat, combine butter and brown sugar, stirring until melted and dissolved (2-3 minutes). Carefully add cream while stirring, bring to a gentle boil, then reduce heat and simmer for 2-3 minutes until thickened.
Remove caramel sauce from heat and stir in vanilla and a pinch of salt. Let cool for 5 minutes. Cut bread pudding into squares and serve warm with generous drizzles of caramel sauce and a scoop of vanilla ice cream if desired.