Footage of Kentaro Kobayashi from the 1990s recently emerged in which he appears to make jokes about the Holocaust.
Japan's Olympic chief Seiko Hashimoto said the video ridiculed "painful facts of history".
The dismissal is the latest in a string of scandals to hit the Games.
Since the start of the year, three other organisers have been forced to step down from Tokyo 2020 - and the games have already been postponed because of the Covid-19 pandemic.
Earlier this week, a composer quit the team creating the ceremony after it emerged he had bullied classmates with disabilities at school
In March, Olympics' creative chief Hiroshi Sasaki quit after suggesting that plus-size comedian Naomi Watanabe could appear as an "Olympig". He later apologised
And in February, Yoshiro Mori was forced to step down as the head of the organising committee after he made remarks about women that were criticised as "inappropriate". Mr Mori was quoted as saying women talked too much and that meetings with many female board directors would "take a lot of time".
This latest scandal has seen former comedian Mr Kobayashi strongly criticised for a sketch he performed 23 years ago, in which he and another comedian pretend to be children's entertainers.
In the sketch Mr Kobayashi turns to his colleague, referring to some paper dolls, saying they are "the ones from that time you said 'let's play the Holocaust'", according to AFP news agency.
Japanese Prime Minister Yoshihide Suga described the comments as "outrageous and unacceptable".
Meanwhile Rabbi Abraham Cooper, of the US-based Holocaust research body Simon Wiesenthal Center, said: "Any person, no matter how creative, does not have the right to mock the victims of the Nazi genocide."
Mr Kobayashi himself has issued a statement responding to his dismissal.
"Entertainment should not make people feel uncomfortable. I understand that my stupid choice of words at that time was wrong, and I regret it," it said.
Olympics chief Seiko Hashimoto said he was not aware of the sketch before Mr Kobayashi's appointment, and apologised for "causing concern to those involved in the Olympics, to the citizens of Tokyo and the Japanese public".
Despite the last-minute change of leadership, Japan's prime minister said the event should proceed as planned - kicking off two weeks of competitive events.