In Japan, the per capita arable land area is small, the natural resources are short, and agricultural production has the typical characteristics of the peasant economy. Even if there are many unfavorable conditions, the level of agricultural modernization of Japan has exceeded that of many developed countries. We conduct comparative analysis on the development characteristics of China's agriculture and Japan's agriculture, and summarize the realization path of Japan's agricultural modernization. Based on the reality of China, we can draw on the following experience from Japan: establishing and improving the agricultural legislation for agriculture; improving the rural cooperative economic organizations; deepening the reform of the agricultural land property right system; increasing the government's direct funds input of into agriculture; improving the agricultural technology extension system.