The comment came after his ministry announced a ban on girls playing rugby or boxing at Tonga High School - one of the main government schools in the country.
Prime Minister 'Akilisi Pohiva has since issued a statement that said all Tongans should be encouraged to participate and there should be no discrimination in any sports.
But the Education Minister Penisimani Fifita is not budging and has blamed any negative reactions on cultural differences.
"So you are talking from a different culture to a different culture," he said.
