Go to Cantler's for
crabs. It's a few minutes away from the tourist area and right on the water.
The USNA visitor center is decent and they have a nice museum. They have a fantastic collection of ship models, if your into that kind ot thing. You don't need any special credentials to walk on base during visitor hours. You will need DOD credentials to drive on. If I remember correctly the pedestrian gate is #1 and it is about a block from the city dock.