Baguette Box has two locations (
http://www.baguettebox.com/) - they're not exactly in downtown, but they're not far and both locations are in really cool neighbourhoods that are ripe for exploring to walk off your meal
For dinner, I'm a huge fan of Serious Pie in downtown (
http://www.tomdouglas.com/restaurants/serious-pie) - best pizza crust I've ever had
I was also recently at Quinn's Pub (
http://quinnspubseattle.com/) in Capitol Hill - it's a gastropub that has a decently sized menu for different appetites - great great food all around
Matt's in the Market in Pike Place (
http://www.mattsinthemarket.com/) is also very well-known for both the location and food. Get a seat upstairs and enjoy one of the best views in town with your lunch
stay the f*ck away from Cheesecake Factory...