The Syrian Observatory for Human Rights said 42 prisoners and 15 militants died when the facility near Mayadin, in Deir al-Zour province, was hit on Monday.
The Deirezzor24 news website said dozens of civilians and Syrian rebel fighters were detained there.
A coalition spokesman said it was looking into the casualty reports.
Mayadin, which lies in the Euphrates river valley about 45km (28 miles) south-east of the city of Deir al-Zour, has been targeted frequently by coalition aircraft.
US intelligence officials believe IS moved most of its leaders to the town in recent months as Iraqi government forces advanced into the Iraqi city of Mosul and a US-backed alliance of Kurdish and Arab fighters encircled the Syrian city of Raqqa.
The Syrian Observatory, a UK-based group that monitors the country's six-year civil war through a network of sources on the ground, said the prison was hit at dawn on Monday by what were believed to have been coalition aircraft.
Deirezzor24, which is run by an activist collective, reported that the facility was located near the village of Tayibiya wa al-Maaharufa, and that it had once been the home of a commander of al-Nusra Front, an al-Qaeda-linked group now known as Hayat Tahrir al-Sham.
Many fighters opposed to IS, including those from Tahrir al-Sham and the Western-backed Free Syrian Army, were held there along with civilians, it said.
It put the death toll at 60, but said it only included two militants serving as guards.
Coalition spokesman Col Ryan Dillon acknowledged that it had conducted strikes on "IS command-and-control elements" in the Mayadin area on Sunday.
"This disrupts [IS's] ability to conduct terrorist attacks in both in Iraq, Syria, against coalition partner forces, and also against our homelands," he told the BBC.
Col Dillon added that the strike was "meticulously planned" and that coalition officials "always account for and take into account human suffering and any type of casualties in our planning."
At the start of June, the coalition said its 21,035 air strikes in Syria and Iraq since 2014 had unintentionally killed at least 484 civilians. However, human rights groups believe the true figure is far higher.
Air wars, an organization that tracks allegations of civilian deaths, estimates that at least 4,118 civilians are likely to have died.