To press on with the Games in Rio, the second most affected city in Brazil by the Zika crisis, would be "irresponsible" and "unethical," the letter argued.
"Our greater concern is for global health. The Brazilian strain of Zika virus harms health in ways that science has not observed before," said the letter, signed by experts in the United States, Britain, Australia, Canada, Norway, the Philippines, Japan, Brazil and South Africa, among others.
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," it said.
"Should that happen to poor, as-yet unaffected places (e.g., most of South Asia and Africa) the suffering can be great."
Zika can cause birth defects, including a devastating syndrome known as microcephaly in which babies are born with unusually small heads and brains.
Nearly 1,300 babies have been born in Brazil with the irreversible defect since the mosquito-borne Zika began circulating there last year.
The World Health Organization and top US public health officials have called on those traveling to Brazil to take precautions against mosquito bites, and have said pregnant women should avoid areas where Zika is circulating, including Rio de Janeiro.
The Olympics and Paralympics, set for August 5 through September 18, "will take place during Brazil's wintertime when there are fewer active mosquitoes and the risk of being bitten is lower," WHO said this month.
And on Thursday, the top US public health official, Tom Frieden, said "there is no public health reason to cancel or delay the Olympic Games."
But the open letter, signed by doctors and researchers at leading universities around the world, expressed worry that officials are not adequately protecting the public against the danger posed by Zika.
"It is unethical to run the risk, just for Games that could proceed anyway, if postponed and/or moved," the letter read.
The letter urged the WHO to "conduct a fresh, evidence-based assessment" of the situation in Brazil and its recommendations for travelers.
WHO has 'conflict of interest'
Given the big financial investments at stake, the letter questioned whether the UN health agency was able to give a non-biased view of the situation.
It said the world body may not be properly considering the options, which include moving the Games to a place where Zika is not present, postponing them until Zika is under control, or cancelling them.
"We are concerned that WHO is rejecting these alternatives because of a conflict of interest," said the letter.
"Specifically, WHO entered into an official partnership with the International Olympic Committee, in a Memorandum of Understanding that remains secret."
It called on the UN health agency to disclose the memo.
"Not doing so casts doubt on WHO's neutrality," it said.
"WHO must revisit the question of Zika and postponing and/or moving the Games. We recommend that WHO convene an independent group to advise it and the IOC in a transparent, evidence-based process in which science, public health, and the spirit of sport come first," the letter said.
"Given the public health and ethical consequences, not doing so is irresponsible."