Jason Lee, a 37-year-old managing director at Goldman Sachs, was charged with first-degree rape for the shocking attack in the playground of the super-rich.
The East Hampton Star, citing sources, reported the incident happened late Monday or early Tuesday during a "pool party" that "involved nudity."
In a statement, police said they were called to Lee's tony Clover Leaf Lane address at around 6:30 a.m. Tuesday because of an unspecified "disturbance."
Home in East Hampton where rape allegedly took place.
"While investigating, officers learned that a 20-year-old woman had been sexually assaulted inside the residence where several people had gathered," the statement said.
Lee's wife was in the courtroom when he was arraigned on Wednesday. He was released on a bail of $20,000, which is $13,000 less than he and his wife reportedly paid to rent the 4-bedroom, 3-bathroom home for a month.
A knowledgeable source said Lee has taken "a leave of absence while he attends to this matter."
A rep for Goldman Sachs declined comment.
Lee's lawyer, Brian Desesa, said, "He vehemently denies these allegations and we look forward to clearing his name."