The singer died on Monday in Los Angeles, the family told the Associated Press.
They described him in a statement as a "solitary man with a heart driven to connect to the world".
"He spoke honestly to people and connected them to each other," the statement said.
Known for his smooth baritone vocals and sumptuous soul arrangements, he wrote some of the 70s best-remembered songs, including Just The Two Of Us, Lovely Day and Use Me.
On Lovely Day, he set the record for the longest sustained note on a US chart hit, holding a high E for 18 seconds.
Although he stopped recording in 1985, his songs remained a major influence on R&B and hip-hop.
His track Grandma's Hands was sampled on Blackstreet's No Diggity, and Eminem reinterpreted Just The Two Of Us on his hit 1997 Bonnie And Clyde.
Lean On Me has recently become associated with the Coronavirus pandemic, with many people posting their own versions to support health workers.
"We are devastated by the loss of our beloved, devoted husband and father," said Withers' family in a statement.