What a Family! Siggi, Naomi, Ivan, Alan, Banking, Philanthropy, a Sex Museum, Race Track & Wilzig Castle

Most people, when they build a mansion in the Hamptons, keep that information private except for family and friends. This was not the case when Ivan Wilzig, the son of a wealthy banker, came to town. He bought the brow of a hill deep in the woods north of Water Mill and on it built a 14,000-square-foot medieval castle. It was complete with a dungeon, throne room, grand staircase and turrets. Arms of the castle enclosed a large courtyard featuring a swimming pool and ornamental fountain.

With the construction completed—this was around 1998—Ivan, later known as “Sir Ivan,” opened the metal gates and invited a few hundred people to an opening day Jungle Masquerade Ball party. He greeted them in a “jungle man” costume, but in later years he was known to appear onstage and at his parties as the superhero Peaceman, singing songs he’d written himself, his glittering cape swinging behind him. One song had even climbed up the charts on Billboard for a few weeks.

By all reports, that first party was a wild, celebrity-filled affair, and what started out as R-rated changed to X-rated in the wee hours. It was the talk of the town for weeks.

The following year, Ivan held his party again, and this time, I wangled an invitation.

Around 10 that night, I drove a group of friends up Deerfield Road toward the castle. And an astonishing thing happened. About a quarter of a mile before the gate, still on Deerfield Road, a man in a business suit standing on the white line in the middle of the street motioned for me to stop. He then leaned into the driver’s window. I saw he was wearing an earpiece.

“May I see your invitation?” he asked.

“I’m just driving up the road.” A lie.

“Well if you’re going to the castle, you’ll need your invitation.”

As he leaned in, one side of his jacket opened slightly and I saw that under it he had a gun in a shoulder holster.

We continued on. All of us had seen the gun. We rolled along now in utter silence.  Then there was another man motioning for us to stop. This time I thought it best to show the invitation.

As we drove on, someone in the back seat spoke.

“Does he have a moat?”

Although we successfully got into the party to drink champagne and eat hors d’oeuvres served by men in white while Wilzig entertained, we didn’t stay that long. I think we were all spooked.

Several years later, I attended a more ordinary party at a private home in Mecox. Most of the people there were bankers or stockbrokers. At one point, I met a handsome man in a business suit—everyone was in business suits—who introduced himself to me as Alan Wilzig.

“Are you related to Ivan?” I asked.

“I’m his brother,” he said.

“Do you live at the castle?” I asked.

“I did at first. We both owned it. And we still do. But I’m selling him my half. I’m nothing like him.”

“I can see that,” I said.

I spent about 15 minutes talking to Alan Wilzig. He’d met a wonderful woman—he showed me a picture of her—and told me he intended to marry her and start a family. He was a one-woman man. As far as work was concerned, his dad owned a chain of banks in New Jersey and he was working with him, he said.

I didn’t ask if his brother worked for his dad, too. But my guess was he didn’t. Ivan was advancing peace in the world. The two of them could not be more different.

I never did get to see Alan Wilzig again. Someone told me he’d gotten married and bought a large mansion upstate. But I felt sure he was very happy.

One year, I think it was around 2002, Ivan did not hold his annual party. And the reason was that his elderly father, Siggi, had wanted the place for himself to enjoy the peace and quiet of Water Mill. But the following year, the party resumed.  And the year after that, their father died.

I went to only one other party at the castle after that. I guess I was more like brother Alan after all.

The annual party continues on to this day. But around 2015, Alan was in the news. I think I read about it in the New York Post. It was a scandal. And there, in the paper, was Alan, wearing a helmet, goggles and a race car driver jacket with motor oil patches on it, standing in front of his racing car.

Alan, after a 6-year legal battle with officials from the upstate Town of Taghkanic, had built the largest privately owned race track in the country on his 274-acre estate in that town, despite the fears by the townspeople that the noise would be heard for miles. And now he was going through a divorce. This was fodder for the Post. Poor little rich boy, obsessed with racing. And now this.

I thought: “What a family! What had Siggi brought into the world?”

And then, just three months ago, a book called Unstoppable: Siggi B. Wilzig’s Astonishing Journey from Auschwitz Survivor and Penniless Immigrant to Wall Street Legend came out. On the cover was Siggi Wilzig at 21, standing on the deck of a boat as it passed the Statue of Liberty. A German Jew, he had survived the death camps of Nazi Germany and now, with only a few dollars in his pocket, would begin his new life shoveling snow in the Bronx and working in a Brooklyn sweatshop. Years later, as a fiery, unstoppable businessman, he’d become a multi-millionaire oil, gas and New Jersey bank tycoon.

I also learned, doing research for this article, that Naomi Wilzig, Alan and Ivan’s mother, was the founder of the World Erotic Art Museum in Miami Beach, FL. And the Wilzig family were, and are generous philanthropists.

And I thought: “What a family!”