This article analyzes the agricultural structure among rice sector, livestock sector, and vegetable sector in Japan, adopting the viewpoint of the movement of main production areas. This article also considers the influences of international market access, production adjustment, cultivated land lease market, land improvement and the logic of the farm household. The livestock and the vegetable sector experienced a structural change including the moves of the main production to areas such as Hokkaido and Kyushu, which had an economic rationality from the geographical point of view. Behind this change, there was stiff market competition from abroad as well as domestic products and improvement of infrastructure in production areas. In contrast, the rice sector maintains production areas widely all over Japan, and there are many small-scale rice farmers that don't aim at profit, while large-scale rice farm managements occupy a constant share of the rice sector of Japan. The issue of the structure of Japanese agriculture depends how the rice sector changes.