A Holy City is the city where a Great Prophet was when he founded a religion. This city then becomes "holy" for that religion, and gets some bonus pressure, besides being the very first city from which the religion starts to spread.
Most of the time, the Holy City for a particular religion is the Capital of the nation which founded it.
The Fall 2013 patch made changes on the Holy Cities of religions so that they cannot be razed if captured. Note, however, that you can in fact unmake a Holy City by using an Inquisitor's Remove Heresy ability on it. This will remove all other religions except the one to which the Inquisitor belongs, and will also remove the Holy City status and its pressure if the Inquisitor does not belong to the religion the Holy City represents.