The Pharisees may have been with him the whole time. Matthew's Gospel says the Sadducees are there too, so it is Pharisees and Sadducees.
Unbelief will always run to some other fortress. As soon as you answer questions, as soon as you explain certain queries that are brought up, as soon as in their case they were shown miracles on a huge scale, they run to some other refuge. ‘Ah, we can’t possibly accept this or believe this, until this even greater problem is solved.’ That is how unbelief works. Of course, they weren’t going to get their larger signs, because it was just wilful unbelief.