Footage has emerged showing the three-storey wooden stand filled with spectators falling at a stadium in El Espinal, Tolima department.
The traditional "corraleja" event involved members of the public entering the ring to engage the bulls.
President-elect Gustavo Petro urged local officials to ban such events.
"I ask mayors not to allow more events involving the death of people or animals," he said.
It was not the first time such an accident had happened, he added.
The dead included two women, a man and a child, Tolima Governor Jose Ricardo Orozco said, and at least 30 people were seriously injured. There are fears that the death toll will rise further.
A bull also escaped from the stadium and caused panic in the municipality.
Local councillor Iván Ferney Rojas said the town's hospital and ambulance services were unable to cope with the number of injured, El Tiempo newspaper reports.
"We need support from ambulances and neighbouring hospitals, many people are still unattended," Mr Rojas is quoted as saying.
Sunday's event was part of celebrations of the popular San Pedro festival.
However Mr Orozco said his regional authority would move to ban the corralejas, saying they were dangerous and cruel to animals.