1. Why most nouns ending -o are masculine, and -a are feminine
Spanish is a Romance language, which means it is descended from Latin. In Latin, nouns fell into 5 groups, depending on how they were declined (i.e. how their suffixes changed depending on case).
The first declension nouns usually end in -a in the nominative singular and are mostly feminine, which is why most words ending -a are feminine in Spanish.
The second and fourth declension nouns usually end -us in the nominative singular and are mostly masculine. Latin words ending -us generally evolved into words ending -o in Spanish, which is why most words ending -o are masculine in Spanish.
2. Why mano bucks the trend
mano comes from Latin manus - one of the few feminine fourth declension nouns, which explains why it ends -o.
However, even within these feminine 4th declension nouns, manus is an exception in Spanish, being the only one to both retain its feminine gender and its (morphologically) expected suffix:
Changed gender
- echo > eco (m)
- porticus > pórtico (m)
- īdūs (pl) > idus (m, pl)
- quīnquātrūs (pl) > quinquatro (m)2
lat
domus > it
duomo > fr
dôme > es
domo (m)
Changed suffix
- fagus > [materia] fāgea > faya > haya (f)
- acus > *acūcula > *acūcla > aguja (f)
- socrus > *socra1 > suegra (f)
- nurus > *nora > nuera (f)
- tribus > tribu (f)
Both
- quercus > quernus > al quernus > al quernus + occus > alcornoque (m)
Neither
The reason mano maintained both of these aspects is probably due to a combination of factors:
- it didn't refer to a female person, and so had no semantic pressure to change (e.g. nuera, suegra)
- it was a very common word
- it didn't gain a diminutive suffix in its evolution
- it didn't refer to a tree3
Notes
1. "Father-in-law" was socer in Latin, which evolved expectedly from the singular accusitive socrum > *socro > suegro, which may have also influenced socrus becoming suegra.
2. 1803 DLE.
3. In addition to the examples presented, fourth declension nouns referring to trees were feminine in Latin as well. These almost exclusively became masculine in Spanish, with the exception of the above fagus > haya:
• pinus > pino (m)
• taxus > tejo (m)
• alnus > alno (m)
• prunus > pruno (m)
• populus > *plōppus > pobo, chopo (m) etc