Earlier World Rugby said it was investigating the claims against Francis Kean, who had been one of eight nominees for the seven positions on the influential Executive Committee.
The Sunday Times reported it had heard a recording of Kean while he was in charge of the Fiji prison service, in which the paper alleged homophobic language was used.
World Rugby said it noted the allegations and took them extremely seriously.
"Rugby is a sport built on strong and inclusive values and World Rugby does not in any way condone any abusive or discriminatory behaviour, as outlined within its by-laws."
The former Navy commander Kean, who was convicted of manslaughter in 2007, had been Fiji Rugby's chairman since 2015.
Earlier the high profile advocacy group, Pacific Rugby Players Welfare, had condemned the nomination of Kean.
This week the group's CEO and former Samoan international, Dan Leo, wrote a damning open letter to World Rugby detailing Keane's political and legal history in Fiji.
World Rugby had now confirmed the Fiji Rugby Union had decided to stand down Kean from the World Rugby Council, therefore withdrawing his candidature for the World Rugby Executive Committee.
The international body said while it was important to stress that any allegations must be validated, following dialogue with World Rugby, the Fiji Rugby Union recognised the seriousness of the allegations made and the need for them to be fully investigated, and that it was in the best interests of the sport that Mr Kean stepped down.
The Fiji Rugby Union's seat on the World Rugby Council will be filled by their CEO, John O'Connor.
There are now seven nominations for seven seats on the World Rugby Executive Committee, which will be confirmed at the 2020 Annual Meeting of the World Rugby Council, held by teleconference on 12 May.