The Wallaroos performance was full of errors, while the Japanese success was based on some outstanding defence.
Both sides scored two tries with Australia having the chance to steal the result late in the game but normally reliable goal kicker Lori Cramer missed a penalty attempt in the 77th minute.
The result will improve the world ranking for the Japanese who are currently ranked 12.
The Japanese also beat Fiji 28-14 earlier in the series.
The Wallaroos sang the national anthem in two languages in a first for them that was warmly embraced by the midweek crowd and the players alike.
Australia will play the Black Ferns in June and August before they meet in their opening game of the World Cup in October.