Tokyo 2020 is now scheduled to run from 23 July to 8 August next year after being delayed because of the coronavirus pandemic.
Health experts have cast doubt on whether the games could be held next summer without a vaccine or effective drugs to treat Covid-19 being found.
When asked if the event could be moved again to 2022, Mori replied: "No."
"In that case, the Olympics will be scrapped," he added.
However, the Tokyo 2020 president said he was confident the rescheduled games would go ahead.
"This Olympics would be much more valuable than any Olympics in the past if we could go ahead with it after winning this battle," he told Nikkan Sports.
"We have to believe this, otherwise our hard work and efforts will not be rewarded."
Japan medical association president Yoshitake Yokokura said on Tuesday it would be "difficult" to hold the Olympics in 2021 without vaccines or effective drugs being developed.
"I am not saying that Japan should or shouldn't host the Olympics, but that it would be difficult to do so," he said.
"If the infections are under control in Japan, it will still be difficult to hold the games unless the pandemic is over in the rest of the world."
Most experts think a vaccine is likely to become available by mid-2021, though others have warned there is no guarantee a vaccine can be successfully developed.
Mori added that the Olympics and Paralympics - which will now run from 24 August to 5 September 2021 - could share opening and closing ceremonies as a way of cutting costs.
"Given that the situation has dramatically changed, we have to review key areas, including the ceremonies," he said.
"It's going to a considerable cut in costs and a big message of victory against the global crisis - but it's not easy."
However, Mori said he had not yet asked the International Olympic Committee and International Paralympic Committee whether they would agree to such a plan, and acknowledged that tickets had already been sold for the separate opening and closing ceremonies.