The men's, women's and wheelchair events are scheduled to take place in England this autumn.
New Zealand Rugby League (NZRL) chief Greg Peters said it was "simply too unsafe" to take part.
Rugby Football League (RFL) chairman Simon Johnson called it a "selfish, parochial and cowardly decision".
"Rugby League World Cup organisers have bent over backwards to offer every assurance to the Australians and the Kiwis," Johnson told BBC Radio 4's Today programme.
"We are very unhappy with this decision and we are not prepared to take it lying down."
NZRL chief Peters said the decision to withdraw was made because the safety and wellbeing of the players and staff "cannot be guaranteed to our satisfaction".
"There are stark differences between how the pandemic is being managed in the UK compared to Australasia, and recent developments have highlighted how quickly things can change.
"The tournament organisers have moved heaven and earth to make this work, so it is not an easy decision, but the Covid-19 situation in the UK shows no sign of improving, and it's simply too unsafe to send teams and staff over."
Australian Rugby League Commission (ARLC) chairman Peter V'landys said: "Not participating in this year's World Cup is not a decision the commission has taken lightly, but we must put the best interests of our players and officials first. Protecting them is our absolute priority.
"In the current environment, the risks to the safety, health and wellbeing of the players and officials travelling from Australia to participate in the tournament this year are insurmountable.
"We have again requested the IRL and Rugby League World Cup consider postponing the event until 2022 to enable all players to participate."
About half of Australia's population has been placed back into lockdown following a spike in Covid-19 cases, with stay-at-home orders now in place in South Australia, Victoria and parts of New South Wales.
That is in sharp contrast to England, with the majority of coronavirus restrictions now lifted despite rising cases.
Organisers only confirmed that the tournament would go ahead earlier this month, although Australia, the holders and 11-time winners, did not sign the participation agreement.
One of the issues previously raised is Covid-19 regulations and quarantine rules for players and staff travelling back to Australia and New Zealand.
Australia's borders are currently closed, so anyone returning has to spend 14 days in government-managed quarantine.
There are reports that Australia's National Rugby League (NRL) teams want the tournament postponed to next year because those regulations mean players would only return to their clubs two weeks before trial games for the 2022 NRL season begin.
Rugby league officials estimate around 400-500 players, staff and team officials from NRL sides, representing several different countries, would be involved in the World Cup.
International Rugby League chair Troy Grant was critical of the decision to withdraw, saying he has been in regular communication with Australia's Rugby League Players Association (RLPA) and has received assurances that a number of international players and coaches are satisfied with the safety arrangements put in place.
"Whilst I can appreciate the ARLC's intent to ensure player safety and welfare, I find it difficult to find the words that adequately describe my disappointment with that decision," he said in a statement.