At a cabinet meeting on Friday the decision was made to withdraw from hosting because of a World Bank report looking at its economic impact.
'Akilisi Pohiva said cabinet decided to withdraw because of the financial implications of hosting the event.
RNZI repots preparations for the games have been beset with controversy, delays, and legal action.
Mr Pohiva said the decision to withdraw is another blow for Tonga but it could not be avoided.
"It may look embarrassing but as far as Tonga is concerned, the economic and financial situation of the country at the moment, there is no other option but to cancel the sports. We are being forced by circumstance, by financial and economic circumstance of the day, to do the right thing and the right thing is to cancel."
He said the cabinet made the decision on the basis of a 2013 World Bank report about the economic impact of the event.
While Mr Pohiva said he intended to write a letter to the Pacific Games Council informing them of the decision, the council said it had not heard from the government about plans to withdraw from hosting the games.
President of the council, Vidhya Lakhan, said all they know was what they had heard through the media.
"There has been no official communication, either to the Games Council or the Pacific Games Association of Tonga, about their intentions," said Mr Lakhan.
"It is already known there is a tripartite contract and if one partner wishes to pull out, for whatever reason, I think it is common courtesy that the other partners would be consulted, advised and we could look at ways and means of keeping the Games in Tonga if there was intention to move it out.
"So we will wait until we hear and then we will do whatever we feel is necessary to ensure that the games remain in Tonga."
"But I just can't believe what I'm hearing because only about two months ago...I think it was in a question and answer session he assured the parliament and the people of Tonga that his government was totally behind the games," he said.
"And the government was supporting it and they will do whatever was necessary to ensure they have a successful games in 2019."
Mr Lakhan said until they were informed otherwise the Pacific Games Council's position is that the 2019 games will be held in Tonga.