Liverpool forward Takumi Minamino gave the four-time Asian champions the lead midway through the first half, before Junya Ito bagged another shortly after the break in Saitama, north of Tokyo.
The Saudis went into the game four points clear of Japan at the top of Group B, knowing a win would seal their place in Qatar.
But Japan's victory instead set up a tense battle for the two automatic qualifying spots, with two games remaining and Australia also still in the hunt.
The Socceroos were in action away to Oman later in the day and drew 2-2.
Japan face Australia in a high-stakes showdown in Sydney on March 24th, before finishing their campaign at home to bottom side Vietnam.