The man was carrying no proof of identity when he was found in the city of Nagato on Saturday.
Police are expected to hand him over to immigration officials who will decide whether he is a genuine defector.
In 2011, nine North Koreans were picked up by the Japanese coastguard after spending five days at sea.
They later settled in South Korea.
Some 29,000 North Koreans have fled the country since the peninsula was divided at the end of the Korean War in the 1950s, many of them crossing the border into China.
In April, a senior North Korean military officer who oversaw spying operations defected - South Korean officials said at the time that such high level defections were rare.
Any North Korean who makes it to the South enters into a rehabilitation programme and is given an aid package to help them start a new life.
Despite this, many find it hard to adjust.