The Horseman is, as usual, the first true cavalry unit in the game, available to all that don't have special earlier units (and have access to Horses, of course). It has very decent Combat Strength for its era, and its mobility conveys it great advantage on the battlefield. Just beware of anti-cavalry units, such as the Spearman - it makes up for having 11 less Combat Strength with its bonus vs. cavalry, and is also available earlier (which means that by the time the Horseman is available, players will have had enough time to raise a sizable army of Spearmen).
Use the Horseman for crushing flanking attacks, to run around the enemy formation and hit at their back, or to harass cities and pillage their improvements and Districts. (Light cavalry even has a special Promotion for pillaging.) Note, however, that Civilization VI doesn't give cavalry an inherent hit-and-run capability, so make sure your Horsemen won't be overwhelmed after attacking.
Before the Iron Age, the role of horsemen was largely filled by light chariots, useful both for reconnaissance and in battle. Though occasionally used by commanders and couriers, the horses of the age were too small to carry much weight and were damn expensive. Around 2000 BC, however, those living in the Central Asian steppes domesticated and cultivated larger, stockier, and hardier breeds. Armed with spears, swords and eventually bows, these horsemen generally raised hell … and changed the face of warfare. Soon enough the Greeks to the west and Chinese to the east adapted mounted warriors for their own purposes, then everyone around did as well. But it was the development of the stirrup attached to a sturdy saddle that made horsemen the dominant shock weapon on the battlefield; the earliest saddles with stirrups date from c. 300 AD, and over the next four centuries the innovation spread across Asia and into Europe.