Coming from the north, I'll add a strong personal endorsement of Skipper's in Darien, GA. Skipper's is on the water, with shrimp and fishing trawlers nearby. Exit 49 (miles north of the Florida state line).
MFF thread on Skipper's
The Sterns recommend Singleton's Seafood Shack in Mayport, FL (4728 Ocean St, Atlantic Beach, FL). The restaurant is a few hundred feet from the tiny car ferry to the north shore. I found the drive along the ocean highway to be remarkably enjoyable. You pass American Beach, with its impressive historical legacy, Amelia Island, Fernandina Beach, etc. The road loops back to I-95 near the border.
They also like Salt Water Cowboys (299 Dondanville Rd, Saint Augustine, FL) on St Augustine Beach. Highway 207 east from I-95, to highway 312 and onto the island. South to the Dondanville road crossing, then right. If you continue south, you can exit and return to I-95, or continue south along the beach road.
JB's Fish Camp in New Smyrna Beach is out on the island, about as far south as the paved road goes. (859 Pompano Ave, New Smyrna Beach, FL). The Sterns suggest the river blue crabs.
I-95 to highway 44 east, and A1A out the south causeway onto the island. I haven't been there, but it looks like a very nice, easy off, ten minutes ride.
Farthest south in the article is Dixie Crossroads in Titusville. 1475 Garden St, Titusville, FL. Garden Street /Florida 406, east off I-95 at exit 220, restaurant is about two point three miles.
From their website:
Dixie Crossroads is famous for shrimp – Reds, Whites, Pinks, Brownies, “Hoppers” and, of course, our trademark specialty…Rock Shrimp, the sweetest little delicacy to ever come from the ocean.