Preheat your oven to 425°F. Line a baking sheet with parchment paper for easy cleanup.
In a large bowl, combine the halved baby potatoes, olive oil, salt, black pepper, garlic powder, and rosemary. Toss until the potatoes are evenly coated.
Spread the potatoes in a single layer on the prepared baking sheet, cut side down. Roast in the preheated oven for 25-30 minutes, or until golden brown and crispy, flipping halfway through for even cooking.
While the potatoes are roasting, mix the lemon zest, lemon juice, and capers in a small bowl. Set aside.
Once the potatoes are crispy and tender, remove them from the oven and drizzle the lemon-caper mixture over the top. Toss gently to combine.
Garnish with fresh parsley before serving.