Muhyiddin Yassin, backed by the corruption-tarnished former governing party, was sworn in by the king.
Mr Mahathir, 94, who has dominated Malaysian politics for decades, described the appointment of Mr Muhyiddin as illegal and a betrayal.
He vowed to seek a vote in parliament to challenge Mr Muhyiddin's support.
Mr Mahathir, who was the world's oldest elected leader, had returned to power in 2018 in a coalition with his old rival Anwar Ibrahim.
In a surprise victory, he ousted then-prime minister Najib Razak, who is on trial charged with taking millions of dollars from a government wealth fund.
Mr Mahathir threw the country's politics into turmoil last weekend when he resigned, breaking his alliance with Mr Anwar.
But the move backfired, as Mr Anwar initially decided to run for prime minister while Mr Muhyiddin built his alliance.
"This is a very strange thing... Losers will form the government - the winners will be in the opposition," Mr Mahathir said, referring to the outcome of the 2018 election.