The work he refers to was the healing of the man that is recorded in chapter five, where he makes it plain also that he is the Son of God and it was performed on the Sabbath day. You were all shocked and horrified, he says, because I healed a man on the Sabbath day.