Petro beat construction magnate Rodolfo Hernandez with an unexpectedly wide margin of some 716,890 votes. The two had been technically tied in polling ahead of the vote.
Petro, a former mayor of capital Bogota and current senator, has pledged to fight inequality with free university education, pension reforms and high taxes on unproductive land. He won 50.5 percent to Hernandez's 47.3 percent.
Petro's proposals - especially a ban on new oil projects - have startled some investors, though he has promised to respect current contracts.
Supporter Alejandro Forero, 40, who uses a wheelchair, cried as results rolled in at the Petro campaign celebration in Bogota.
"Finally, thank God. I know he will be a good president and he will help those of us who are least privileged. This is going to change for the better," said Forero, who is unemployed.
This campaign was Petro's third presidential bid and his victory adds the Andean nation to a list of Latin American countries that have elected progressives in recent years.
Petro, 62, said he was tortured by the military when he was detained for his involvement with the guerrillas, and his potential victory has high-ranking armed forces officials bracing for change.
Petro's running mate Francia Marquez, a single mother and former housekeeper, will be the country's first Afro-Colombian woman vice-president.
"Today I'm voting for my daughter - she turned 15 two weeks ago and asked for just one gift: that I vote for Petro," said security guard Pedro Vargas, 48, in Bogota's southwest on Sunday morning.
"I hope this man fulfills the hopes of my daughter, she has a lot of faith in his promises," added Vargas, who said he never votes.
Petro has also pledged to fully implement a 2016 peace deal with FARC rebels and seek talks with the still-active ELN guerrillas.
He had raised doubts about the integrity of the count after irregularities in congressional tallies in March, and earlier on Sunday urged voters to check their ballots for any extraneous marks which could invalidate them.
Hernandez, who served as mayor of Bucaramanga, was a surprise contender in the run-off and has promised to shrink government and to finance social programmes by stopping corruption.
He has also pledged to provide free narcotics to addicts in an effort to combat drug trafficking.
Despite his anti-graft rhetoric, Hernandez is under a corruption investigation himself over allegations he intervened in a trash management tender to benefit a company his son lobbied for. He has denied wrongdoing.
Defence Minister Diego Molano told journalists on Sunday afternoon that the killing of an electoral volunteer in Guapi, Cauca province, was under investigation.
Sixty voting locations had to be moved because of heavy rains in some parts of the country, the registrar said.