Twitter chairman Bret Taylor said on the micro-blogging platform that the board planned to pursue legal action to enforce the merger agreement.
"The Twitter Board is committed to closing the transaction on the price and terms agreed upon with Mr Musk...," he wrote.
In a filing, Musk's lawyers said Twitter had failed or refused to respond to multiple requests for information on fake or spam accounts on the platform, which is fundamental to the company's business performance.
"Twitter is in material breach of multiple provisions of that agreement, appears to have made false and misleading representations upon which Mr Musk relied when entering into the merger agreement," the filing said.
Musk also said he was walking away because Twitter fired high-ranking executives and one-third of the talent acquisition team, breaching Twitter's obligation to "preserve substantially intact the material components of its current business organization."
Shares of Twitter were down 6 percent at $US34.58 in extended trading. That is 36 percent below the $US54.20 per share Musk agreed to buy Twitter for in April.
Twitter's shares surged after Musk took a stake in the company in early April, shielding it from a deep stock market sell-off that slammed other social media platforms.
But after he agreed on 25 April to buy Twitter, the stock within a matter of days began to fall as investors speculated Musk might walk away from the deal. With its tumble after the bell on Friday, Twitter was trading at its lowest since March.
The announcement is another twist in a will-he-won't-he saga after the world's richest person clinched the multi-billion-dollar deal for Twitter in April but then put the buyout on hold until the social media company proved that spam bots accounted for less than 5 percent of its total users.
The contract calls for Musk to pay Twitter a $US1 billion ($NZ1.6b) break-up if he cannot complete the deal for reasons such as the acquisition financing falling through or regulators blocking the deal. The break-up fee would not be applicable, however, if Musk terminates the deal on his own.
Musk's decision is likely to result in a long protracted legal tussle between the billionaire and the 16-year-old San Francisco-based company.
Daniel Ives, an analyst at Wedbush, said Musk's filing was bad news for Twitter.
"This is a disaster scenario for Twitter and its Board as now the company will battle Musk in an elongated court battle to recoup the deal and/or the breakup fee of $1 billion at a minimum," he wrote in a note to clients.