He demanded that the regional government clarify whether it now considered itself independent following a speech by Catalan president Carles Puigdemont on Tuesday night.
This requirement is a necessary step before triggering Article 155 of the constitution, which would allow Madrid to suspend the region's political autonomy.
Rajoy's move could deepen the confrontation between Madrid and Catalonia but it also signals a way out of Spain's biggest political crisis since a failed military coup in 1981.
The prime minister would be likely to call a snap regional election after activating the constitutional mechanism allowing him to do so.
Puigdemont made a symbolic declaration of independence from Spain on Tuesday night but then immediately suspended it and called for talks with the Madrid government.
"The cabinet has agreed this morning to formally request the Catalan government to confirm whether it has declared the independence of Catalonia, regardless of the deliberate confusion created over its implementation," Rajoy said in a televised address after a cabinet meeting to consider the government's response.
Without giving a specific deadline for the Catalan government to reply, Rajoy said: "The answer from the Catalan president will determine future events, in the next few days."
It is not yet clear if and when the Catalan government would answer the requirement but it now faces a conundrum, political analysts say.
If Puigdemont says he did declare independence, the government would likely trigger Article 155. If he says he did not declare it, then far-left party CUP would likely withdraw its support to his minority government.
"Rajoy has two objectives: if Puigdemont remains ambiguous, the pro-independence movement will get more fragmented; if Puigdemont insists on defending independence then Rajoy will be able to apply Article 155," said Antonio Barroso, deputy director of London-based research firm Teneo Intelligence.
"Either way Rajoy's aim would be to first restore the rule of law in Catalonia and this could at some point lead to early elections in the region".
Puigdemont had been widely expected to unilaterally declare Catalonia's independence on Tuesday after the Catalan government said 90 percent of Catalans had voted for a breakaway in an Oct. 1 referendum that Spain had declared illegal and which most opponents of independence boycotted.
Madrid responded angrily to Puigdemont's speech, saying the Catalan government could not act on the results of the referendum.
"Neither Mr. Puigdemont nor anyone else can claim, without returning to legality and democracy, to impose mediation... Dialogue between democrats takes place within the law," Deputy Prime Minister Soraya Saenz de Santamaria said.
Invoking Article 155 to ease Spain's worst political crisis in four decades would make prospects of a negotiated solution to the Catalonia crisis even more remote.
Some of Catalonia's largest companies have moved their head offices out of the region and others were set to follow if he had declared independence.
15:46 Catalan leader 'close declare independence from Spain any minute' despite Madrid's warnings
23:40 Catalonia: a dangerous confrontation8
17:11 Catalan leaders given five-day deadline to declare independence4
17:06 Spanish government give Catalan leaders five-day deadline to clarify 'declaration of independence'1