The US has blocked the $1.2bn (£880m) sale of money transfer firm Moneygram to China's Ant Financial, the digital payments arm of Alibaba
It is the highest profile Chinese deal to be rejected by Washington since Donald Trump came to power.
Regulators overseeing foreign investments in the US had refused to support the takeover, the firms said.
The geopolitical environment had "changed considerably" since the merger was announced last year, they added.
'Disappointed'
The collapse is a blow to the ambitions of Alibaba's billionaire executive chairman Jack Ma, who had promised President Trump that he would create a million US jobs.
Alibaba, which owns Ant Financial together with Alibaba executives, saw the US market as a way to expand overseas in the face of fierce domestic competition from the likes of Tencent's WeChat.
But in a joint statement on Tuesday, Ant Financial and Moneygram said they had abandoned the deal after failing to "obtain the required approval for the transaction from the Committee on Foreign Investment in the United States, despite extensive efforts to address the Committee's concerns".
A spokesman for the Treasury Department, which chairs the committee, said it is barred from discussing individual deals.
The committee is charged with reviewing deals for national security implications.
Moneygram chief executive Alex Holmes said he was disappointed by the outcome and noted the "geopolitical environment has changed considerably" in the year since the deal was announced.
Deals blocked
The US has been toughening its stance on business dealings with China.
The country launched a formal review of China's intellectual property practices last year. US politicians and military leaders have also urged the administration to take a closer look at Chinese investments in America, particularly in the technology industry.
In September, the US blocked the $1.3bn sale of US Lattice Semiconductor to Chinese-backed Canyon Bridge Capital Partners, citing concerns over the "potential transfer" of intellectual property from Lattice, which makes advanced computer chips.
Other deals that have been frustrated by US objections include China Oceanwide Holdings Group's $2.7bn purchase of US life insurer Genworth Financial, and Chinese buyout firm Orient Hontai Capital's $1.4bn acquisition of US mobile marketing firm AppLovin.
In this case, US authorities may have been worried about protecting the personal data handled by the financial companies, said Scott Kennedy, director of the project on Chinese business and political economy at the Center for Strategic and International Studies in Washington.
"China doesn't have a reputation of providing a great deal of protection for privacy and private data," he said.
"It may be that Alibaba is sort of caught in the dilemma of being a company from China .... It's very difficult to shake that off."
Mr Kennedy said he thinks the deal was likely to have been scrutinised even in a different political environment.
But it is also true that US lawmakers from both parties are raising increased objections to Chinese investments, he said.
"This case ... needs to be analysed on its own independent merits, but the broader trend is definitely in a tighter direction," he said. "There's a growing Washington consensus that China plays unfair."