The US, UK, Australia and Canada will not send government representatives to the Games because of concerns over China's human rights record.
This includes widespread allegations of abuse against the Uyghur minority group.
France, host of the next Summer Games, said it would not join the boycott.
The Winter Olympics are set to take place in Beijing in February.
"The United States, Britain and Australia have used the Olympics platform for political manipulation," Wang Wenbin, a spokesperson at the Chinese foreign ministry, said.