The 31-year-old came to Brazil as the most decorated Olympian in history and has now won 23 medals at five Games.
He swam the second leg for the USA, who won in a time of three minutes 9.92 seconds, with France second in 3:10.53 and Australia third in 3:11.37.
Phelps came out of retirement in 2014, having quit after London 2012.
He has won twice as many Olympic gold medals as the athlete second on the list, former Soviet gymnast Larisa Latynina.
"On the block, I thought my heart was going to explode," said the record-breaking American. "I was so hyped, so excited."