Both Australia's goals came in the first half, with New Zealand scoring in injury time at the end of the second half through debutant Gabi Rennie, who headed the ball into the net just minutes after coming on as a replacement.
The Football Ferns were one of four teams, including the US, Sweden and Great Britain, that took a knee before their game as a part of the global call for greater racial equality.
New Zealand also have the world champion United States and Sweden in their group, with the World No.5 Swedes stunning the the US 3-0 in their match last night.
The Football Ferns will have to beat at least one of those teams to advance to the knockout stage.
Tameka Yallop opened the scoring for the Australians in the 20th minute at Tokyo Stadium, thumping a shot past Erin Nayler.
Captain Sam Kerr doubled the lead in the 33rd minute, with her header glancing off the underside of the crossbar.
A bright spark for the Football Ferns, however, was the arrival of Rennie on the international scene, marking her debut with a goal just minutes after taking to the pitch.
Football Ferns head coach Tom Sermanni paid tribute to the young forward.
"It's great for any player to score a goal but to come out to the Olympics, first time, obviously very excited to make the squad, probably unsure if she was going to make the squad, and to then come on and score her first international goal, it's a special moment for her."
While defeat wasn't the desired result, the Football Ferns playing their first competitive game since March 2020, thanks to the impact of the Covid-19 pandemic, is important for the side going forward, head coach Tom Sermanni said.
"Getting today's game under our belts was of benefit to us - it helps us to be more focused and understanding of what to expect in the next game."
Their next game against the US on Saturday night was a challenge the side needed to be prepared for, captain Ali Riley said, , especially given the US loss to Sweden.
"It's a very quick turnaround and I think we need to be very specific with our preparation and our analysis [of tonight]. You can't spend too much time thinking about this game because we need to prepare for the next one.
"We're going to have to be ready for a US team that is not used to losing two games in a row. I'm not sure if they ever have, so I think they're going to want revenge and we will be that target."