The man was carrying no proof of identity when he was found in the city of Nagato on Saturday.
Police are expected to hand him over to immigration officials who will decide whether he is a genuine defector.
In 2011, nine North Koreans were picked up by the Japanese coastguard after spending five days at sea.
They later settled in South Korea.
Some 29,000 North Koreans have fled the country since the peninsula was divided at the end of the Korean War in the 1950s, many of them crossing the border into China.
