Preheat oven to 350°F (175°C) with rack in center. Grease the sides of a 9-inch cake pan (not the bottom) or use a 10-inch cast iron skillet.
Melt 4 tablespoons of butter in the pan on the stovetop over medium-low heat. Remove from heat and sprinkle ¾ cup brown sugar and ½ teaspoon cinnamon evenly over the butter; do not stir.
Peel peaches, slice into ½-inch wedges, and arrange tightly in a circular, overlapping pattern over the brown sugar layer.
Whisk together flour, baking powder, salt, cinnamon, and nutmeg in a medium bowl. Set aside.
Beat softened butter and granulated sugar on medium-high speed for 3-4 minutes until pale and fluffy. Scrape down the bowl as needed.
Add eggs one at a time, beating well after each. Mix in vanilla extract and scrape down the bowl.
Alternately add the flour mixture and milk to the butter mixture, starting and ending with flour. Mix on low until just combined, being careful not to overmix.
Gently dollop the batter over the peaches and spread into an even layer, being careful not to disturb the fruit.
Bake for 40-48 minutes until golden brown and a toothpick inserted in the cake center comes out with moist crumbs. Tent with foil if browning too quickly.
Let cake rest in the pan for 10-15 minutes. Run a knife around the edges, place a serving plate on top, and confidently flip. Lift the pan and rearrange any stuck peaches.