The Pharisees were already deeply prejudiced against the Lord, but there was something more to this complaint. They were personally offended by what they saw.
‘For ye see your calling, brethren,’ says the apostle, ‘how that not many wise men after the flesh, not many mighty, not many noble, are called: 27 But God hath chosen the foolish things of the world to confound the wise; and God hath chosen the weak things of the world to confound the things which are mighty (1 Corinthians 1:26-27). We are to try to reach all but we are told here that much of the fruit of the gospel will come from the lowly, so of course we must associate with them.