Just a short jump from NYC is City Island in the Bronx. Check out this Gothamist article, which gives a pretty good overview. The island is only 1.5 miles long, so you can easily walk it. Water access is difficult, but we went to the Harlem Yacht Club (don’t have to be a member) and sat on the deck overlooking the harbor and had a cocktail. We also had a drink in The Snug on City Island Avenue (surprisingly good Bloody Mary) and chatted with a local man (a true clam digger) who gave us a little townie flavor.
There are also a few quirky antique and curiosity shops that are worth a visit. Explore it for yourself. It’s a great NY adventure.