Fumio Kishida said he had called China's ambassador to protest against the "incursions".
On Friday, about 230 Chinese fishing boats and coast guard vessels sailed near islands claimed by both countries.
Beijing has been increasingly assertive about waters it believes are Chinese.
The Japan-controlled, uninhabited islands - known as the Senkaku in Japan and the Diaoyu in China - are the source of a long-running dispute.
The Japanese coast guard said on Monday that about 13 Chinese coast guard ships, some of them armed, had been seen near the islands, higher than the usual number.
"The situation surrounding the Japan-China relationship is significantly deteriorating," Mr Kishida told Cheng Yonghua, Beijing's envoy to Tokyo, according to a statement on the foreign ministry website.
"We cannot accept that [China] is taking actions that unilaterally raise tensions."
Mr Kishida said he had lodged repeated protests since Friday about the "incursion and violation of our sovereignty". He said China should withdraw its official vessels immediately.
Mr Cheng reiterated China's claim over the area at the meeting, he told reporters, and asked for both countries to continue to have dialogue.
He also said the increase in number of coast guard ships was to oversee the increased fishing in the area by Chinese boats, according to AP news agency.
"Please understand that it's an effort by the Chinese side to avoid further complications to the situation," AP quoted Mr Cheng as saying.
In July, China rejected an international tribunal ruling that its claims have no rights in the South China Sea, raising tensions with neighbours who have competing claims in the area.
The case was brought to the Permanent Court of Arbitration by the Philippines, which has seen disputes flare up with China over fishing grounds near the Scarborough Shoal and Spratly Islands in particular.
China has indicated it is open to talks with the Philippines outside the tribunal ruling to settle the dispute.
On Tuesday, former Philippines president Fidel Ramos, chosen by Philippines President Rodrigo Duterte as envoy to pave the way for talks, was in Hong Kong meeting with Beijing-linked contacts.