The 56-year-old was a successful banker before his foray into politics, and will join the board today, before stepping up to become chair early next year.
Sir John, who led the country for eight years between 2008 and 2016, will replace John Judge, who has been the bank's chair for more than five years.
ANZ Group chair David Gonski said Sir John's previous experience is what made him such a desirable candidate for the role.
"Sir John Key's strong international career in banking and his understanding of and contacts across the Asia-Pacific - where many Australian and New Zealand companies are increasingly trading - will add great value to the governance of ANZ," he said.
Sir John was the Head of Global Foreign Exchange with Merrill Lynch, and a member of the Foreign Exchange Committee of the New York Federal Reserve Bank, before going into politics.
Mr Judge will bring his time at the company to a close in January next year, before handing the reins to Sir John.
Mr Gonski described Mr Judge's tenure as one that encompassed "significant change" at the company.
"The National Bank and ANZ brands and technology systems were successfully merged in 2012 and the business has grown significantly," he said.
"ANZ New Zealand now has the biggest market share for home loans, deposits, credit cards and KiwiSaver, and is leading the way with digital services."