Tokyo 2020 is now scheduled to run from 23 July to 8 August next year after being delayed because of the coronavirus pandemic.
Health experts have cast doubt on whether the games could be held next summer without a vaccine or effective drugs to treat Covid-19 being found.
When asked if the event could be moved again to 2022, Mori replied: "No."
"In that case, the Olympics will be scrapped," he added.
However, the Tokyo 2020 president said he was confident the rescheduled games would go ahead.