My heart belongs to Newport, the historic little gem of a city on Aquidneck Island in Rhode Island, but to say so is a little misleading because my Newport also includes Middletown, which borders it to the east and north. I have known both places for more than half a century, and in my mind they are inextricably, intimately, eternally bound together, one and the same. I love them for any number of reasons, the most important of which are that I was blissfully happy there as a boy, my family's roots there are deep and the waterscape there is surpassingly, indescribably beautiful.

Until my wife and I went to Newport for a couple of days in the spring of this year, my Newport had existed solely in memory for almost exactly two decades. My mother died in her own bed in Middletown in September 1986, and soon thereafter her four children decided that our ailing and disoriented father needed full-time care in a nursing home. I drove him to New Hampshire in March 1987 and returned to Newport only once more: a few months later, to interview friends of my parents' for a book I was writing about them. That was it. We sold our parents' house, and in so doing forfeited the right, cherished by all of us, to be called Newporters.

If this gets you thinking about Vanderbilts and Lorillards and Belmonts, about the famous "cottages" built on the Newport shoreline by the super-rich in the late 19th and early 20th centuries, think again. That's not my Newport. Wretched excess is not my cup of tea, and my parents never had the wherewithal to indulge in excess of any kind. In the early 1950s friends who owned a small house in Middletown suggested that my parents might like to rent a bungalow there for summer vacation. They loved Middletown, and in 1955 were able to buy a couple of acres (no view of the water except in winter) and build a real cottage there. They did it all for about $25,000, my mother's bequest from her parents, which seemed to Bill and Helen Yardley a small fortune.

They were actually the second Yardleys to build on Aquidneck Island, a familial fact of which my father, an ardent totemist, was inordinately proud. In 1882 his grandmother, Jane Woolsey Yardley, was suddenly widowed in Connecticut. She decided to build a house for herself and her four children next door to her mother in Newport, and she then did something that to this day I find absolutely extraordinary: she went to Cornelius Vanderbilt II and asked for a loan of $5,000. He granted it, apparently immediately, "without interest," and within a year she repaid it in full. I have Vanderbilt's cordial letter releasing her from the loan. But that's not the end of it. In May 2007, touring Vanderbilt's mansion the Breakers with my wife, in its gift shop I found a copy of the American Institute of Architects' guide to Newport; pictured inside, to my astonishment and delight, was the "Jane Yardley House" at 91 Rhode Island Avenue.

It has always struck me as odd that my family of church mice somehow found its way onto the fringes of this citadel of American wealth and power. But what strikes me as even odder is that none of us seems to have resented our impecuniousness. I guess we were just too glad to be there to carry chips on our shoulders. Certainly that is true of me. As a boy I spent six mostly unhappy years as a scholarship student at two different boarding schools, building up an antipathy toward the rich that is with me to this day, but I've never felt that I didn't belong in Newport. My parents couldn't possibly have afforded membership in the Casino, the venerable tennis club, or Bailey's Beach, where the wealthy sunned and swam, but we weren't a tennis-playing family, and we could sun and swim as much as we wanted at Second or Third Beach in Middletown—which are, in fact, much better beaches than crummy old Bailey's.

As to the Casino, in July 1954, when I was 14 years old, I was allowed onto its grounds for something a whole lot better than tennis: the First American Jazz Festival, subsequently (and universally) known as the Newport Jazz Festival. I was in the early stages of what was to become a lifelong passion for jazz, and it simply blew my mind—no other phrase will do—that staid old Newport had become a temple of jazz. I was present for almost every minute of every festival for the next decade—in 1955 it moved from the Casino to Freebody Park—including the evening I will never forget: July 7, 1956, when Duke Ellington's nonpareil orchestra exploded in a "Diminuendo and Crescendo in Blue" that turned the audience into an exuberant, roaring throng, put the Duke on the cover of Time (with me clearly visible in a crowd photo inside) and gave me a cherished, indelible memory.

Two years later a gifted if self-infatuated filmmaker named Bert Stern came to Newport and made Jazz on a Summer's Day. It's not a perfect depiction of the festival—inexplicably, he left out some marvelous performances and included some ordinary ones—but it is the most vivid portrait of Newport that's ever been done. Stern interwove shots of the festival with others of Newport itself, most notably aerial views of the America's Cup yacht competition that took place in Rhode Island Sound that summer. Nowhere will you see Newport's magnificent rocky coast and impossibly blue water with more breathtaking clarity than in this film.

Nowhere, that is to say, unless you're actually there. Revisiting Newport and Middletown this May for the first time in too many years, I realized more than ever before that it is the water that defines the place for me. This is a little strange, as I am not a water person as that's usually understood: I'm not much of a sailor, gave up swimming years ago and stay away from the beach on dermatologist's orders. Yet I can look at water for hours. Two years ago my wife and I bought an apartment high above the Pacific coast in Peru, her native country. I can sit there forever, just watching the water roll in, thinking about the next stop to the southwest (Easter Island) and about my boyhood fictional hero, Horatio Hornblower, sailing up that same coast to do battle against the demented tyrant, El Supremo.

The water off Peru is beautiful, but the water with which Newport is surrounded almost surpasses belief. It is the deepest, darkest blue I have ever seen, indeed at times it seems almost violently so. If in Peru I dream of Easter Island, in Newport I dream of whalers out from New Bedford, of ships smashing against rocks, of—as the hymn of my boyhood put it—"those in peril on the sea." This isn't morbid, it merely draws upon Newport's historic role as seaport, as a place where the American nation was founded and shaped, where, as my wife pointed out, innumerable houses in the old center of town have widow's walks.

When I was a boy, Newport was still a working sailor's town. Thames Street was filled with bars and rough trade of just about every imaginable variety. Now Thames Street is a shopping mall for day-trippers, and the old harbor is packed with fancy yachts. One of the reasons I stayed away from Newport for so many years was that I heard more than I cared to about its transformation from old colonial seaport to contemporary tourist trap. There's plenty of that as well as plenty of new construction. Some of the old mansions along Bellevue Avenue and Ocean Drive have been converted to condos, and the shopping malls along Route 214 in Middletown are strictly Anywhere, U.S.A.

Yes, that's depressing enough, thank you, but to my surprise it all pretty much rolls right off my back. My Newport may be a little harder to find now, under all the ticky tack, but it's still there: the Redwood Library and Athenaeum, to which my family made pilgrimages almost every day; Trinity Church, with its perfect colonial steeple and its unique high pulpit from which my father was privileged to preach on a few occasions; the house on Indian Avenue, different now (and better!), but tended by its present owners with as much love as Bill and Helen Yardley lavished on it; St. George's School, where for several summers I worked as a day-camp counselor, with its deep green lawn rolling down toward the deep blue water of Sachuest Bay.

As I write these words I have beside me a satellite photograph of southern Narragansett Bay. There is more water in it than land. Aquidneck Island floats in the middle like a large ship at permanent mooring, with nothing more substantial than a few slender bridges tying it to the North American continent. Almost everything I see in that photograph stirs a memory, and almost all those memories are happy. If it is my fate to have Newport nowhere except in memory for the rest of my life, I will still count myself fortunate.

Jonathan Yardley is the book critic of the Washington Post. His memoir Our Kind of People is partly set in Newport.