The Brazilian authorities stopped Gunnar Bentz and Jack Conger from boarding the flight at Rio de Janeiro airport, bound for the US, for questioning late on Wednesday.
Two teammates - Ryan Lochte and James Feigen - were barred from leaving the country, but Lochte had already left.
Police have queried their accounts of a robbery they reported in Rio on Sunday.
The four swimmers said they had been robbed at gunpoint in a taxi in Rio.
But police said their account of when they arrived at the Olympic Village did not square with CCTV recordings.
Lochte's lawyer, Jeffrey Ostrow, said he got back to the US two days ago.
"He arrived back before the judge issued anything," he said.
"He was never asked to remain for further investigation or for any other purpose after he met with Brazilian authorities after he gave a statement."
Lochte is one of the most successful swimmers in history, with 12 Olympic medals. He once had his own reality television show in the US.
In Rio, he swam in two events, winning gold in the 4x200m freestyle relay.
Feigen, who told the San Antonio Express he was still in Brazil, won gold in the 4x100m freestyle relay.
Accounts of what happened to the swimmers have been confusing from the beginning.
Lochte first told US TV network NBC about it but International Olympic Committee spokesman Mark Adams said reports of the robbery were "absolutely not true".
The US Olympic Committee later confirmed the swimmers' accounts.
Police question swimmers' claims
Lochte himself told police that he and teammates Feigen, Bentz and Conger had all attended a party in the French Olympic team's hospitality house on Saturday evening.
He said that during their return drive to the Olympic Village, their cab was stopped by men posing as police officers.
One of the men held a gun to his head, Lochte said, and robbed him of his money and personal belongings.
But police investigating the case say they have found no evidence of the robbery. They also point to what they say are inconsistencies in the swimmers' accounts.
Lochte and Feigen told police they and their fellow swimmers had returned to the Olympic Village at 4am.
But CCTV recordings appear to show the swimmers returning to the Olympic Village at 7am.
Police said they had not been able to track down the driver who the swimmers said drove them back to the village.
They also said that Feigen and Lochte gave different accounts of how many men robbed them.
Investigators have requested a search warrant for the rooms the swimmers stayed in with a view to examining Feigen's mobile phone.