A Self-Guided Sag Harbor Village Architectural Tour

While today’s rich and famous tend to build their mansions behind hedges and outside of public view, the rich and famous of Sag Harbor’s bygone whaling days built right on the main streets. And boy did they have style!

Take an informative walk down Main Street with this tour of some the most historic and architecturally significant buildings in the village. Then relax at a more modern restaurant with a bottle of rosé—or two.

East Side of Main Street, North to South

AMERICAN HOTEL
ca. 1845, 45 Main Street The hotel is on the site of the earliest house built in Sag Harbor, James Howell’s Inn, where Lieut. Col. Jonathan Meigs captured British officers in their beds during the American Revolution. (A monument to Meigs stands west of 44 Union Street.) After the great Sag Harbor Fire of 1845, the cabinetmaker Nathaniel Tinker rebuilt the site as his shop and warehouse. When the building was converted to a hotel in 1877, the porch with Gothic columns was added to the front.

MUNICIPAL BUILDING
ca. 1846, 45 Main Street Originally constructed in 1846 as the Mansion House Hotel, the building was converted to the Union School in 1871 and, after the completion of Pierson High School in 1907, to the Village Hall.

PELEG LATHAM HOUSE
ca. 1790, 117 Main Street The Peleg Latham House is one of two Federal-style structures at the juncture of Madison and Main. It was designed with a residence above and a commercial story at street level, as was typical for a Sag Harbor merchant’s house of the period.

CIVIL WAR MONUMENT
Main Street at Madison Street Dedicated in 1896, Sag Harbor’s Civil War memorial includes a cannon, a statue of a Union soldier, and a planter that was originally a horse trough.

STANTON HOUSE
1840, Corner of Madison and Main Streets The Stanton House is an archetype of Greek Revival architecture: gable ends face the street, doors had both transom and side lights and pilasters on the side. In 1853, Admiral Stanton accompanied Commodore Matthew Perry on his historic visit to Japan, which opened up Japan to the West. 

BENJAMIN HOPE HOUSE
1860, 165 Madison Street The Hope House is a rare example of Victorian architecture in Sag Harbor, with a mansard roof in the Second Empire style. After the end of the whaling era, money was scarce in the town. Benjamin Franklin Hope was a skilled clockmaker.

SYBIL DOUGLAS HOUSE
ca. 1790, 189 Main Street This five-bay Georgian house was built for Benjamin Huntting, who sent out the first successful whaling ship to Brazil in 1785. Originally on the site of what is now the Sag Harbor Whaling and Historical Museum, the structure was sold to Capt. Douglas and moved in 1838, when his son, Benjamin Huntting II, decided he wanted a grander home.

JOHN JERMAIN MEMORIAL LIBRARY
1910, 201 Main Street In the Beaux Arts Classical Revival style, the John Jermain Memorial Library was donated to the village in 1910 by Mrs. Russell Sage in honor of her grandfather. With its dome covered in Guastavino tile, it’s impressive. Recently, the building was conserved, enlarged and modernized. 

VAN SCOY HOUSE
ca. 1810, Corner of Main and Jefferson Streets A gambrel roof with an ingenious arrangement of windows underneath is characteristic of buildings associated with Benjamin Glover, Sag Harbor’s builder-carpenter. The house is named for Arnold Van Scoy, a Daguerreotype photographer who lived there in the 1850s.

GREAT WAR MEMORIAL
Corner of Main and Jermain Streets The Great War Memorial at Otter Pond features a large boulder with an eagle perched atop it and a bronze plaque: “1917–1919: In honor of the men of Sag Harbor who served in the Great War and in memory of those who gave their lives.”

West Side of Main Street, South to North

L’HOMMEDIEU HOUSE
ca. 1840, 258 Main Street The L’Hommedieu House would look right at home in Greenwich Village. Samuel L’Hommedieu, grandson of a Huguenot fugitive, operated “a rope walk on Glover Street;” that is, a long, narrow building where workers walked back and forth twisting hemp into rope.

BENJAMIN HUNTTING HOUSE
ca. 1845, 200 Main Street The Benjamin Huntting House is a fine example of the Greek Revival style. Benjamin Huntting II hired architect Minard Lafever, who designed the Old Whalers’ Church, to design a classic temple front mansion. It later became the summer home of Mrs. Russell Sage and then the Sag Harbor Whaling & Historical Museum, in 1936. Note the blubber-spade decoration along the roofline.

CUSTOMS HOUSE
ca. 1790, 912 Main Street Sag Harbor was one of the first American seaports to be designated an official Port of Entry back in 1789. The Custom House was not only the workplace of U.S. Custom Master Henry Packer Dering, but also the home where his nine children were raised.

HANNIBAL FRENCH HOUSE
186 Main Street In 1860, the Hannibal French House was redesigned and enlarged from a modest Federal dwelling into an elaborate Italianate design attributed to architect Minard Lafever. Despite the decline in the whaling industry, the French family fortune remained. Note the birdhouse under the eaves.

THE SAG HARBOR HISTORICAL SOCIETY–THE ANNIE COOPER BOYD HOUSE
ca. 1796, 174 Main Street Originally, this was a Colonial half-house built by Joseph Whitting Foster. The northern half was added c. 1802. In 1871 William Cooper Jr. bought it for rental income. His daughter Annie Cooper Boyd, who grew up in Sag Harbor, was given this house as a marital summer home. Starting in 1904, Annie added dormers, more windows and a porch. Annie’s daughter Nancy Boyd Willey willed it to the Sag Harbor Historical Society in 1998.

SITE OF CINEMA
90 Main Street The landmark Sag Harbor Cinema opened as the Elite Theatre on November 1, 1919. It was destroyed by fire in December, 2016. The Sag Harbor Partnership is in the process of rebuilding it as the Sag Harbor Cinema Arts Center.

Other Buildings of Interest

WATCHCASE
15 Church Street In 1880 the Business Aid Committee of Sag Harbor persuaded entrepreneur Joseph Fahys to relocate his watchcase factory from New Jersey to Sag Harbor. The Fahys factory provided 400 well-paying jobs. It closed in 1931. In 1937 Bulova Watch Company located a subsidiary there. That closed in 1980. Various proposals for the buildings were mooted over the years, and nothing happened until the early 2000s, when work began to convert it into condos. Watchcase opened for sales beginning in 2013.

OLD WHALERS’ CHURCH
44 Union Street Built in 1843-44 and designed by noted architect Minard Lafever, the exterior is the best example of the Egyptian Revival style extant in the United States. Its decoration includes a blubber-spade motif along the roofline. Originally, the church boasted a 185-foot steeple visible to ships rounding Montauk Point, but it lifted off intact during the 1938 hurricane and smashed into the Old Burying Ground. 

OLD BURYING GROUND
44 Union Street The Old Burying Ground was a village cemetery, with the first interments being the infant children of innkeeper James Howell, buried in 1767. The Old Burying Ground is the resting place of Sag Harbor’s early residents; the last person to be buried here was Hetty Parker, who died in 1870.

BROKEN MAST MONUMENT
Oakland Cemetery, Jermain Avenue Sculpted by Robert Eberhard Launitz in 1856, it reads: “To commemorate that noble enterprise the whale fishery and a tribute of lasting respect to those bold and enterprising ship mast sons of Southampton who periled their lives in a daring profession and perished in actual encounter with the monsters of the deep. Entombed in the ocean. They live in our memory.”

THE WINDMILL
1966, Long Wharf This replica windmill, named after John A. Ward, Jr., who spearheaded its erection, was built in 1966. It replaces one built in 1760, which stood 50 feet to the west of the current site but which is no longer extant. A plaque commemorates local writer John Steinbeck’s contributions to the annual Whalers Festival and to the building of the windmill.

Find more information on the best places to tour, and to live, at BehindtheHedges.com.