Ocean tiles do not provide much for a city, and almost never have any resources unlike the Coast. Fortunately (since you cannot found a city in the water, like in Civilization: Beyond Earth: Rising Tide), you usually won't need to worry about that, because you will have few or no Ocean tiles close enough for a city to work.
For all naval and embarked units, oceans are impassible until the Cartography technology is researched (or for Norway, the Shipbuilding tech, and for Maori, no tech). Thus, they present an insurmountable obstacle to early game exploration, and it is usual for different continents to remain separate (without contact with each other) until someone researches the necessary tech to sail in ocean.
The deep waters of the oceans scared mankind for much of history, braved only by the likes of the Polynesians and Vikings. But, once Europeans began sailing across them and colonizing everything, the world began to shrink ... until today, when people cross them without trepidation.