The singer died on Monday in Los Angeles, the family told the Associated Press.
They described him in a statement as a "solitary man with a heart driven to connect to the world".
"He spoke honestly to people and connected them to each other," the statement said.
Known for his smooth baritone vocals and sumptuous soul arrangements, he wrote some of the 70s best-remembered songs, including Just The Two Of Us, Lovely Day and Use Me.
On Lovely Day, he set the record for the longest sustained note on a US chart hit, holding a high E for 18 seconds.
Although he stopped recording in 1985, his songs remained a major influence on R&B and hip-hop.
His track Grandma's Hands was sampled on Blackstreet's No Diggity, and Eminem reinterpreted Just The Two Of Us on his hit 1997 Bonnie And Clyde.
Lean On Me has recently become associated with the Coronavirus pandemic, with many people posting their own versions to support health workers.
"We are devastated by the loss of our beloved, devoted husband and father," said Withers' family in a statement.
"With his poetry and music, he spoke honestly to people and connected them to each other.
"As private a life as he lived, close to intimate family and friends, his music forever belongs to the world. In this difficult time, we pray his music offers comfort and entertainment as fans hold tight to loved ones."
US musician Chance the Rapper led tributes, describing the singer as "the greatest" and recalling some of his own personal memories.
Rock star and actor Lenny Kravitz posted that his "voice, songs, and total expression gave us love, hope, and strength".