A Holy City is the city where a Great Prophet was when he founded a religion. This city then becomes "holy" for that religion, and gets some bonus pressure, besides being the very first city from which the religion starts to spread.
The Fall 2013 patch made changes on the Holy Cities of religions so that they cannot be razed if captured. Note, however, that you can in fact unmake a Holy City by using an Inquisitor's Remove Heresy ability on it. This will remove all other religions except the one to which the Inquisitor belongs, and will also remove the Holy City status and its pressure if the Inquisitor does not belong to the religion the Holy City represents.