Sylvester Manor: Historic East End Home with Slave Cemetery Is Open

Everyone on Shelter Island remembers Alice Fiske. She was a no-nonsense lady of high breeding who considered herself the Lady of the Manor. And she was the Lady of the Manor. She lived on the biggest estate in town, Sylvester Manor, a property of several hundred acres with a main house that faced out onto Gardiner’s Creek. She’d go off on her daily chores in her red Cadillac on some days and in her white Cadillac others, depending upon her mood and outfit. You couldn’t miss her. She wore white gloves, and a sunhat, sometimes a classic white kitchen bonnet worn by servants in the kitchen in colonial times to keep sparks from the stove from setting their hair on fire.

Alice had a loud, commanding voice. You’d hear her coming. And you’d hear her going. She often talked about the Manor, about the steep slave staircase alongside the kitchen, about the pump house, the engine barn, the big military cannon someone dug up on the front lawn in 1950, and the windmill. She said there was a cemetery on the property where hundreds of slaves were buried.

As for socializing with others, Alice knew just about everybody in town, but she also knew rich friends in other parts of the country who had large mansions. They’d sometimes come to visit. On rare occasions, she’d visit them. Sylvester Manor had a large white wooden gate that sat on Route 114 opposite where that road makes a 90 degree turn from the town center to head north toward Shelter Island Heights. Usually it was closed, unless she was expecting guests. But sometimes she’d have it opened wide to welcome the general public attending events or meetings held there for many of the charitable organizations she supported.

By the early years of this current century, Alice Fiske began to show signs of age. She was in her 80s. In the end, the lord takes us all. What would become of Sylvester Manor? She had no children. It had been willed by her late husband to her nephew in California, years earlier, a man who was a founding member of Pixar. What would he do?

What many people did know about the Manor was its astonishing history. It had been built by Nathanial Sylvester early in the colonial era at a time when he and his brother Constant were in the sugar cane business. Constant owned a sugar plantation in Barbados. Nathanial owned the farm plantation on Shelter Island. Nearby, the towns of Southold and Southampton, our first settlements, had recently been founded. (They were founded in 1640.) East Hampton was founded in 1648. The Sylvesters, rich English traders and merchants, took ownership of the entire island of Shelter Island in 1652. And there they built what can only be called a plantation as a “staging” area for the plantation in the Islands.

There were, at the time, no other English people on the island. But there were, in abundance, Manhansett Indians sleeping in encampments in the woods. They worked with the Sylvesters, in many cases as indentured servants, for a long time, helping in the raising of sheep and cattle and horses and pigs, toiling in the fields growing vegetables, building furniture and fences and other things the Sylvesters asked be made from the wood in the forests to send down to the sugar plantation. And they were soon joined by dozens of black African slaves, brought north from the Caribbean aboard some of the Sylvester’s merchant boats.

Out buildings were built, also warehouses, barns and tool sheds and haylofts. Things made and grown there were shipped out because some of the things needed in Barbados were not available on that island and could not be grown or constructed there. This 8,000-acre island in New York was the backup to that island in the Caribbean.

It’s not hard to imagine how life was on Shelter Island in those early years. The slaves did what they were told. There were African traditions. The Indians cooked around campfires at the end of the day. Native songs were sung. Meanwhile, in the Manor itself, Nathaniel Sylvester and his wife, Grissel and their children lived, and later so did grandchildren.

But there was a problem. Both of the Sylvesters and their wives were Quakers. They had been so before they left England. This was a scandalous new religious group that was, at the time, anathema to the Anglican beliefs of many of their fellow Englishmen.

George Fox had founded that faith in England in the late 1640s. Then he came to America, too. Why sit and listen to a minister read from a prayer book? We are all God’s children. We sit in a circle as friends, and when the Lord moves us, we speak of the beautiful world that we live in and the joy it gives us, and the thankfulness we feel to have found ourselves in it.

In other parts of the New England colonies, the Puritans were enraged by the presence of the Quakers. They would arrest them, call them witches and put them to death. The Salem witch trials were largely about the Quaker faith. It was only when legal protections were given to the Quakers that their persecution ended.

George Fox, according to his own written accounts, visited Shelter Island twice, once in the 1650s and again in 1672, when, according to his journal, “I had a meeting at Shelter Island among the Indians, and the king and his council, with about 100 Indians with him….They sat about two hours and I spoke to them by an interpreter, that was an Indian that could speak English very well and they appeared very loving, and they said all was truth, and did make a confession after the meeting of it.”

In November 1659, a vocal adherent of Fox, Mary Dyer, stayed at the Sylvester Manor as a safe haven from those who wanted to do her in. In the end, in 1660, she decided on martyrdom. She traveled to Boston, and there she was hanged.

As it happened, Constant Sylvester died in 1671, and then Nathanial, in 1680. Nathanial left a will. He wanted the plantation to stay intact, and he ordered that his heirs “never sell to strangers.” Within 20 years of his death, with the Barbados sugar plantations failing, they did exactly that. They sold off huge tracts of the island to the Havens family and to the Nichols family. And then, counting their money, many of the heirs left Shelter Island to live their lives elsewhere.

The plantation did continue, however. But now it was just a grand farm, one of several on the island, as other families arrived and built homes on that island. As for the Manhansset Indians, as elsewhere, they died out because they became ill with the white men’s diseases. The slaves continued on, but there were fewer and fewer of them. In a count made in 1776, there were 14 slaves at the Havens manor house, 10 at Nichols and 4 at the Sylvester Manor. The last slave at the Sylvester Manor was “London,” who was emancipated in 1821.

Over the centuries, the Sylvester Manor was handed down from generation to generation, largely through the female side of the family. And so, over the centuries, it has been headed up by the Deerings, the Gardiners, then the Horsfords and the Fiskes. Correspondences with the wealthy, and visits from them, continued. There are letters Thomas Jefferson wrote from his plantation in Virginia to the Deerings. In the 19th century, the Manor became a salon where intellectual figures were invited to come. They included Whittier and Longfellow. In 1908, the architect Henry Bacon renovated the home.

Alice Fiske died in 2006 at the age of 88. The heir to the Manor, Eban Fiske Ostby, was happy in California, but wanted “to find the right thing to do with it,” he told The New York Times. He had no intention of developing it. As it happened, Ostby called upon another one of the Sylvester descendants, his nephew, a young man named Bennett Konesni, who was a musician and writer. Why don’t you take it? Ostby asked.

Konesni had worked on an archeological dig on the property, at the Quail Hill Farm in East Hampton, and even, for four months, at an organic farm in New Zealand. Konesni said he didn’t want to own it, either, but that he did have ideas for it. If it could be worked out, he wanted that it be a public treasure, a working farm and a historic site. Between the two of them, they made a plan.

The Sylvester Manor Educational Farm was created and Konesni made the first director. Cara Loritz is the current executive director, Sara Gordon is the strategic director, and now, four years later, its vision has become a reality. What a stunning development. Sylvester Manor is saved. And the nonprofit that’s now running it is slowly fixing it up and bringing it back to life. In the meantime, on a sporadic basis, it is open to the public.

Was there really a slave cemetery? Alice Fiske had marked a clearing with a boulder where she said the slaves were buried. A crew from the University of Massachusetts came to the island with what they called Ground Penetrating Radar, and there are thought to be up to 200 people buried where Fiske said they were.

What was the story of the big cannon that had been dug up in 1950? It was determined to have been built in 1671 by the British Navy. Was it here for a reason? The Sylvesters and the Barbados plantation were in full swing. The Sylvesters owned sailing ships to bring foodstuffs and furniture and fence posts down, and slaves back to Shelter Island. There were pirates out there. Surely a cannon was needed. As for its having been buried, during the third Anglo-Dutch War in 1674, Dutch soldiers circled the house, and it might have been a good idea to bury it to avoid the English connection.

On September 21, my wife and I attended an “open house” on the property. Down by the water, several local Shinnecock Indians in full regalia were cooking food, chanting songs and performing rituals similar to those the Manhassetts might have done in the 1600s. They called their program “A Celebration of The Three Sisters: Corn, Beans and Squash,” which, along with chicken, were cooked over a roaring fire and eaten by those attending.

We took a tour of the house led by Richard Hlavacek, a docent, and we went through the front parlor, the west parlor, the library and the dining room. It’s all as it was, a private home, now run-down but in essentially working condition.

In the West Parlor, there’s a painting on the wall of a six-year-old girl. She sits at a table next to a blue porcelain dish. One local resident, who had been in the house doing work for Alice Fiske, once pointed out that in a glass breakfront on the opposite wall, there’s the same blue porcelain dish as in the painting done centuries ago.

Paintings elsewhere in the house mark the busy life of this family. There’s another painting in the West Parlor of Ezra L’Hommedieu, a wealthy Frenchman who was part of the family. There’s a portrait in the library of Eustace Von Liebling, a well-known German teacher. A family member had gone to Germany to study chemistry under his tutelage one year. There on the wall now was his teacher.

In the foyer hangs a painting of Alice Fiske, the most recent and last of the paintings made of family members. She poses in front of the Manor House in her signature bonnet. One of her Cadillacs is in the background.

Across from her painting, on the opposite wall of the foyer, there’s a false wall that with the turn of a key opens up to reveal a huge steel and concrete walk-in safe. When it was finally opened, it was found to contain nearly 10,000 business records, letters and family papers. They are all now at New York University in New York, where they have been sorted and catalogued into a remarkable
collection.

The Sylvester Manor is open to the public from time to time during this interim period, including a house tour on November 23. For their full schedule, go to sylvestermanor.org. You owe it to yourself to see this centuries-old treasure, revealed for scholars and visitors from all over the globe.

BACK TO Dan Rattiner's Stories