The 61-year-old New Zealander, appointed in January 2020, has left for personal reasons.
"I'm disappointed to be leaving at this time," said Cotter, who led Scotland to the World Cup quarter-finals in 2015.
"The team has made some significant improvements over the last few years both on and off the field."
Cotter is the fourth international head coach to leave his role in the past two months, following the departures of Wayne Pivac (Wales), Eddie Jones (England) and Dave Rennie (Australia).