The ordinal number for 53 is quincuagésimo tercero, as you've found. In Spanish each part of the cardinal number (for example, 53, cincuenta y tres) is turned into its corresponding ordinal, only without the y (if there was one); if the cardinal is a single word but two figures, you must imagine them separated (veintidós → veinte y dos).
Some of the smaller ordinal numbers have irregular and/or alternative forms; for example, for 11 once you have both undécimo and the more modern regular form décimo primero (and J. L. Borges famously used yet another form, onceno). The parts of the ordinals that are made up of tens and units, up to 29, can be found as two words (vigésimo quinto) or one (vigesimoquinto); the tens+units above 29 are all written as two words (trigésimo primero, never *trigesimoprimero). This arbitrary rule, incidentally, follows the one about whether you can write a two-figure number as one (veinticinco) or two (treinta y uno).
As you already guessed, many of the less obvious forms are basically never used; any educated speaker will understand quincuagésimo as "fiftieth" but far fewer will get noningentésimo as "nine hundredth". The modern tendency is to make up ordinals in a regular way using a suffix, -avo (feminine -ava, plural -avos, -avas) that is/was usually reserved for fractions. That is, to say onceavo, doceavo, treceavo, ..., cincuentaitresavo, cincuentaicuatroavo, etc. Since ordinals and fractions are not prone to be mixed up, people find this OK.
There are additional complications with numbers higher than two or three figures, but for the most part these are rarely found in speech, and in writing they are usually left as figures plus the ordinal signs (masculine º, feminine ª). When someone needs to say one of these out loud it's not uncommon to use the bare cardinal number (for example, you'll hear journalists saying el doscientos cuarenta y dos aniversario de la Revolución Americana, instead of the proper el ducentésimo cuadragésimo segundo aniversario...).
