Of course the Jews always identified the Zion mentioned in Scripture with the earthly Jerusalem. In doing this they had to explain how sometimes the prophets pronounce a blessing on Israel and at other times a curse. Paul here solves the problem by showing what the words of blessing apply to, and so he robs the Judaizers of yet another plank of their argument, for by their own claim they belonged to the earthly Jerusalem, and yet the blessing of God is upon a quite different Jerusalem, which belongs to a quite a different covenant. He shows that it is Christians who have been set free from the law, who are the true inheritors of the promise, not those who still cling to an earthly Jerusalem. It is Jerusalem above which is the ‘mother’ of all who have faith, not Jerusalem below. In saying that this he makes the promise of blessing through the Abrahamic covenant extend to Gentile believers also. This is such an important point that he next finds support for it by turning to the Old Testament.
The name Zion was initially applied to the Jebusite fortress within Jerusalem taken by David (2 Samuel 5:6-7), but later extended and used as a name for the whole city of Jerusalem (2 Kings 19:21, 31; Psalm 51:18; Isaiah 2:3). It is called the City of God (Psalm 87:1), because it was the place where God had promised to dwell with his people (Psalm 9:11; 69:35; 76:2; 99:2; 102:16; 128:5; 147:12; Isaiah 2:3; 14:32; 28:16; 30:19; etc.). But alongside all these references are others that speak of Zion as an earthly place about which the Lord complains (Isaiah 3:16; 4:3-4; 33:14; etc.).
How can any think that Paul is simply describing the state of affairs in his own day and imagine that he cherished the hope that the earthly Jerusalem would come out of the doldrums into which it had currently fallen? It could not be plainer that he is describing the essential nature of these two cities. There is a Jerusalem above and a Jerusalem below. The one is heavenly in origin, a picture of the church; the other is earthly and, although it was the capital of the land which God blessed under the old covenant, it never ceased to be a place of sin and rebellion. Jerusalem below was the capital of a nation into which men and women entered by natural birth; Jerusalem above is the holy city of which a person can only become a citizen by the new birth. Those who are born again from among both Jews and Gentiles have citizenship of this heavenly Jerusalem even while they are still living in mortal bodies on earth. They shall be changed in the twinkling of an eye, at the last trump, and given glorified bodies suitable for the world to come.