Why most nouns ending -o are masculine, and -a are feminine
Spanish is a Romance language, which means it is descended from Latin. In Latin, nouns fell into 5 groups, depending on how they were declined (i.e. how their suffixes changed depending on case).
The first declension nouns usually end in -a in the nominative singular and are mostly feminine, which is why most words ending -a are feminine in Spanish.
The second and fourth declension nouns usually end -us in the nominative singular and are mostly masculine. Latin words ending -us generally evolved into words ending -o in Spanish, which is why most words ending -o are masculine in Spanish.
Why mano bucks the trend
As Richard says, mano comes from Latin manus - one of the few feminine fourth declension nouns, which explains why it ends -o.
However, even within these feminine 4th declension nouns, manus is an exception in Spanish, being the only one to both retain its feminine gender and its (morphologically) expected suffix:
- echo > eco (m)
- porticus > pórtico (m)
- īdūs (pl) > idus (m, pl)
- quīnquātrūs (pl) > quinquatro (m)2
- domus > duomo
it > dôme
fr > domo (m)
- acus > *acūcula > *acūcla > aguja (f)
- socrus > *socra1 > suegra (f)
- nurus > *nora > nuera (f)
- tribus > tribu (f)
- fagus > [materia] fāgea > faya > haya (f)
- quercus > quernus > al quernus > al quernus + occus > alcornoque (m)
The reason mano maintained both of these aspects is probably due to a combination of factors:
- it didn't refer to a female person, and so had no semantic pressure to change (e.g. nuera, suegra)
- it was a very common word
- it didn't gain a diminutive suffix in its evolution
- it didn't refer to a tree3
1. "Father-in-law" was socer in Latin, which evolved expectedly from the singular accusitive socrum > *socro > suegro, which may have also influenced socrus becoming suegra.
2. 1803 DLE.
3. In addition to the examples presented, fourth declension nouns referring to trees were feminine in Latin as well. These almost exclusively became masculine in Spanish, with the exception of the above fagus > haya:
- alnus > alno (m)
- prunus > pruno (m)
- pinus > pino (m)
- taxus > tejo (m)
- populus > *plōppus > pobo, chopo (m) etc
Appendix: classification of feminine -o words
- From Catalan: nao, seo
- Latin irregular: mano, ratio, libido
- Female persons: virago
- Abbreviations: demo (demostración), dinamo (máquina dinamoeléctrica ¿?), disco (discoteca), eco (ecografía), foto (fotografía), furgo (furgoneta), hijadalgo (hija de algo), moto (motocicleta), polio (poliomielitis), porno (pornografía), quimio (quimioterapia), radio (radiodifusión), sado (sadomasoquista), seño (señorita), sinhueso (from "sin hueso"), telefoto (tele + fotografía)
- Times: [las horas] uno, cuatro, cino, ocho
- Letters: [la letra] o, ro
- Misc: la Gestapo, la segundo, la magneto