Jews of Sag Harbor: An Expert Explores the Story of a Special Place

Below is a condensed version of a presentation by Karl Grossman at a program organized by the Sag Harbor Historical Museum and held at Temple Adas Israel.

Sag Harbor has long been a center for Jews on the East End of Long Island. Jews came to the town in substantial numbers after Joseph Fahys opened a watchcase factory here in 1881. He recruited Jewish engravers from Hungary where engraving was a fine art among Jews. He or his representatives would go to Ellis Island and bring them straight to Sag Harbor.

My grandfather, Herman Grossman, came to Sag Harbor in 1902 to work at Fahys. He met Stefanie Spiegel, also a Hungarian Jew. Stefanie’s sister, Ernestine, had married into the Spitz family, Hungarian Jews who owned Spitz’s Furniture & Appliance Store on Main Street in Sag Harbor. Members of the family also worked at Fahys.

Temple Adas Israel, which describes itself as the oldest synagogue on Long Island, was founded in 1896 as Temple Mishcan Israel. It changed its name in 1948 when it became Conservative after many years as an Orthodox congregation. My wife Janet and I are members of Adas Israel, where she was long a member and secretary of its board.

In the museum here, which Janet arranged, are the synagogue’s original memorial plaques, on one of which is the name of my grandaunt Ernestine Spitz, who died in 1928. Now, a wrinkle in the Sag Harbor Jewish story for those who might be unfamiliar with it: the deep division for years between the Hungarian Jews and the Jews of Sag Harbor with roots in Russia and other European countries. I explore this divide at the start of a TV documentary I made as chief investigative reporter at WVVH-TV, The (Unusual) Jewish History of Sag Harbor, which can be viewed on YouTube.

When Hungarian Jews began arriving in Sag Harbor to work at Fahys’ watchcase factory, they didn’t get along with other Jews already living in the town. In her well-researched work, Yankees in Yarmulkes: Small-Town Jewish Life in Eastern Long Island (1986), Helene Gerard, a librarian from Eastport, writes that in 1890 the Jewish Cemetery Association of Sag Harbor was formed “with six trustees, all of them Russian Jews. This is significant,” she goes on, because in Sag Harbor there were two distinct Jewish communities.

“The Jews from Russia and those from Hungary brought with them European social antagonisms and refused to worship, socialize or be buried together,” Gerard explains.

And so, the Hungarians formed their own Independent Jews Cemetery Association of Sag Harbor in 1899, right next to the Russian Jewish cemetery. “A fence … was erected between the two, which still exists,” writes Gerard.

In the documentary, I comment, “How dopey it was,” that this division between Jews in Sag Harbor existed, especially because Jews weren’t particularly welcome in neighboring East Hampton and Southampton.

Hungarian Jews might have thought themselves better than Russian and Polish Jews, but for the Nazis, there was no distinction — they killed them all, including the families of Herman and Stefanie. On my mother’s side in Poland, all were murdered, too.

In 1918, Gerard relates, “the (Sag Harbor) Jewish community had financial difficulties, and the synagogue’s mortgage was foreclosed. Three members of the congregation bought it back at public auction,” and later that year a “new certificate of incorporation was filed. From the names listed,” she relates, it is clear that by then — after two decades — “the Russian and Hungarian factions had finally made peace and joined in one congregation.”

The division between the cemeteries still exists, but the descendants of the Hungarian and Polish Jews are divided no more. One cemetery is owned and run by Temple Adas Israel, and the other owned and run under an independent name.

In terms of Jewish numbers in Sag Harbor, Gerard cites a 1902 Sag Harbor Express article reporting “a recent census indicating a total population of 3,438,” then continues: “Our population is cosmopolitan … there are about 650 adults of foreign birth. … The Hebrew invasion, which is comparatively recent, sums up, men, women and children, about 500.”

Thus, nearly 15% of Sag Harbor’s population early in the last century were Jews, which is remarkable.

The Sag Harbor Express story went on to note that when Jews arrived in the town from Ellis Island by steamer, they were greeted at the docks by townspeople crying, “Jerusalem is coming! Jerusalem is here!”

In East Hampton, southeast of Sag Harbor, no one was proclaiming, “Jerusalem is here!” There were very few Jews. Indeed, a 2009 story in The Southampton Press to mark the 50th anniversary of the Jewish Center of the Hamptons reported that as late as the 1950s, “The East Hampton Jewish community consisted of perhaps 11 families, most of whom ran local businesses.”

It was at East Hampton’s exclusive Maidstone Club that Groucho Marx, after being told that while he could play golf there as a guest, he wouldn’t be allowed to become a member because he was a Jew, famously commented, “My kids are only half-Jewish. Can they play at least the front nine?” (There are several versions of this story.)

Deserving much credit for breaking the discriminatory floodgates in East Hampton was the late Evan Frankel, a scrappy, tough guy originally from the Lower East Side. I vividly recall Frankel telling me how he had become “the sole Jew living south of Montauk Highway in East Hampton.” And on an 11-acre estate, no less. South of Montauk Highway was, and remains, the most upscale part of the village.

Frankel had come to the East End to build radar facilities during World War II. Steel was in short supply, and he devised a way to use wood to build the frames on which the radar equipment was perched.

Some of the few Jews in East Hampton had come to him in the 1950s, he related, wanting “to build a little shul.” Frankel characteristically thought in a big way. He bought the stately Borden Estate on Montauk Highway, the main entrance to East Hampton village, and the four acres on which it sits. Frankel, to whom establishing Jewish presence in East Hampton was very important, said it showed that “Jews had arrived” in the town, and he donated the building and land for what became the Jewish Center of the Hamptons. He worked closely with another wealthy Jew, J.M. Kaplan, whose summer house in East Hampton is now the Center for Conservation of The Nature Conservancy.

Great architectural and spiritual splendor was to come to JCOH with a new sanctuary named Gates of the Grove. Frankel was central to getting Norman Jaffe, the renowned architect from Bridgehampton, to design it. Paul Goldberger, architecture writer for The New York Times and The New Yorker, has described Gates of the Grove as a “truly sacred space,” Jaffe’s “greatest work.”

It is, indeed, a masterpiece.

To Sag Harbor’s southwest, in Southampton, the Jewish population also had been small. But there’s now also the Chabad of Southampton Jewish Center, which notes it is “the first synagogue in historic Southampton since its 1640 founding …”

“You can imagine,” related a story in New York Magazine, “the consternation when … a family arrived that clearly hadn’t seen the membership brochure. The man … had a bird’s nest of a beard and wore dark suits all day long. He and his wife were young but had a bevy of children — two at first, five before long. They invited guests on Saturday morning — dozens of them.” The reference is to Rabbi Rafe and Chany Konikov.

The story went on: “Almost immediately, Southamptonites began whispering, ‘Who are these people? What do they want? What do we do about them?’ … Soon enough complaints were registered. Letters were sent … Finally the villagers did what Hamptonites have done before them in the face of unprecedented change: They sued.”

It was quite a struggle for the Chabad.

An excellent book about what it was like to be a Jew in Southampton is Together But Apart: The Jewish Experience in the Hamptons. Published in 1966, it was written by Abe Frank, born “when there were about 20 Jewish families in Southampton,” he relates, and raised there, going on to graduate from Columbia Law School, becoming an attorney, and then an administrative law judge for the National Labor Relations Board in Washington, D.C.

During World War II, he was in the U.S. Army and was with the 120th Evacuation Hospital when it entered Buchenwald Concentration Camp after it was liberated in April 1945. He provides an eyewitness account of the horror in Together But Apart.

“Was there anti-Semitism in Southampton?” Frank writes. “Of course. It lurked in hidden corners of hostile minds, waiting to confront us, usually on unequal terms … like a constant pressure on the nerves, a part of day-to-day living, and we put up with it, as an old man puts up with a ringing in the ears.”

For Jews in Southampton, “We had to accept there were people who did not like Jews, any Jews … And there were times when the knife cut truly deep … No way around it. In Southampton we were different from other people … We had a sort of prescience as to who was anti-Semitic and who was not. People put out vibes, sometimes unintentionally, and our antennae were always quick to catch them. …” Frank continues, “While there was obviously no welcome mat for Jews in Southampton, not everyone bore a grudge against the family of Israel and Rachel, and there were many who showed generosity and kindness to the Israelite in their midst.”

For Temple Adas Israel in Sag Harbor, a major turning point came when Leon Morris, the last of several part-time rabbis, became its first full-time rabbi in 2010. In 2014, Rabbi Leon and his family made Aliyah, and he’s now president of the Pardes Institute in Israel. He was succeeded by Rabbi Dan Geffen.

Another major turning point has been the restoration of the synagogue, a magnificent job that started in 2021. It originally was a vision of Rabbi Leon. Architect Lee Pomeroy drew up plans but, sadly, passed away and was succeeded by another superb architect, Bill Chaleff. Overseeing the project was contractor Roy “Buddy” Wines and his Southampton-based company RLW4. There were only minor changes in the footprint of the synagogue. The exquisite sanctuary has been left largely as it was. The stunning stained-glass windows fashioned by Sag Harbor artist Romany Kramoris, who used 10,000 pieces of glass, received vital protection. Most of the renovation involved digging out a basement, expanding other portions of the building and making such improvements as adding an elevator for the disabled and creating space for the Ronald S. Lauder Center for Jewish Learning.

Ronald and Jo Carole Lauder, who live in Wainscott, are members. He is president of the World Jewish Congress and former U.S. ambassador to Austria. His late mother, makeup queen Estee Lauder, was an Adas Israel member. The matching Lauder grant was crucial to the congregation being able to afford the $7.3 million project.

The congregation had homes away from home during the renovation, thanks to the big-heartedness of the First Presbyterian Church in Sag Harbor, known as the Old Whalers’ Church, and the Cormaria Retreat Center. Services were held inside the church and also on its front lawn. And a string of services was held on the lawn of the Cormaria Retreat Center of the Religious of the Sacred Heart of Mary, a global Roman Catholic community.

Sag Harbor has a tradition of accepting others. There’s now another synagogue in Sag Harbor, the Center for Jewish Life, a Chabad. The idea for the center emerged in 2013 in the living room of the home of Rabbi Berel Lerman and wife Brocha, in North Haven, and at the Sag Harbor Inn, and is now on Long Island Avenue and seeking a permanent home.

In 2013, The Jerusalem Post published a story headlined: “The Jewish Hamptons.” It began by telling how the Hamptons have “attracted” many Jews who have “established thriving institutions.”

In Riverhead there’s the Conservative Temple Israel of Riverhead. Its first synagogue building opened in 1926. An extensive history of the shul online includes, among its many pages, these items:

“(In the) 1930s, Sam Saxtein inadvertently stocks the Peconic River with carp when he digs a trench to link his pond to the river. Prior to the trench, there were a healthy number of fresh pike (pickerel) and perch (whitefish) in the Peconic, but no carp, a necessary ingredient for gefilte fish. So, Sam puts in netting at the end of the trench to prevent his carp from escaping. The roe slips through the net. Thanks to Sam, there’s carp swimming in the river today.” (Who knew the Peconic River was a source for gefilte fish!)

“(In) 1937, Nazi youths march on the streets of Camp Siegfied in nearby Yaphank. The Scheinberg family is vigilant. They hide a cache of diamonds in their basement in the event the Nazis come into power and the family needs to flee.”

Here in Suffolk County, a concern by a family that it might need to flee to escape violent anti-Semitism is a concern shared by Jews throughout the world for ages.

There are now plenty of Jews on the East End, and the community is flourishing.

In the Town of Southampton, the top-elected official, the town supervisor, has been until this year — he was term-limited — Jay Schneiderman, whose family are members of Temple Adas Israel. His son Ruben was bar mitzvahed at the synagogue, and before that his daughter, Magda, was bat mitzvahed there. Schneiderman’s late mom, also named Magda, left Hungary for the U.S. in 1938. “Hitler wiped out the rest of her family,” says Schneiderman.

And in Southampton Village, Jesse Warren, a Jew, served as mayor between 2019 and 2023.

A few blocks from the old Fahys’ watchcase factory, now converted into an apartment complex, our son Adam went to elementary school, then to Pierson High School. Adam went on to law school, became Riverhead town attorney, and was a member and then chairman of the Southampton Town Zoning Board from 2002 to 2023 when he was appointed a Southampton Town justice. His great grandparents, Herman and Stefanie, would be so proud of Adam — as we are.

Karl Grossman is professor of journalism at the State University of New York at Old Westbury and host of the nationally aired television program Enviro Close-Up.