Small, who won the World Cup with South Africa in 1995 beating the All Blacks in a gripping final that went to extra time, was 50.
The South African Rugby Union said Small suffered a suspected heart attack late on Tuesday in Johannesburg.
"We are saddened to learn that Springboks great and 1995 Rugby World Cup winner James Small has passed away. A fierce competitor on the field and great friend off it, James will be missed by the entire global rugby community," the All Blacks tweeted.
