Old magazine articles resurfaced in which Keigo Oyamada, 52, boasted of the bullying and said he had no regrets.
After the scandal broke, the performer apologised. But the outrage in Japan has now led to his resignation.
It is the latest setback for the games, which are opening on Friday under the shadow of the coronavirus outbreak.
"I have become painfully aware that accepting the offer of my musical participation in the Tokyo 2020 Olympics and Paralympics lacked consideration to a lot of people," said Mr Oyamada, who is widely known as Cornelius.
In a statement, the Tokyo 2020 organising committee said Mr Oyamada's actions were "absolutely unacceptable".
"In light of his sincere apology, we expressed a willingness to allow Mr Oyamada to continue his work on preparations in the short time remaining before the opening ceremony," it said.
However, the committee later decided to accept his resignation and apologised for the "offence and confusion caused".
The organisers of the Tokyo games have had to deal with several high-profile resignations.
In March, creative chief Hiroshi Sasaki quit after suggesting that a female comedian could appear as an "Olympig".
Mr Sasaki said plus-size entertainer Naomi Watanabe could wear pig ears at the opening ceremony. He later apologised.
In February, Yoshiro Mori was forced to step down as the head of the organising committee after he made remarks about women that were criticised as "inappropriate".
Mr Mori was quoted as saying women talked too much and that meetings with many female board directors would "take a lot of time".