Mi is an adjetivo posesivo (like tu, su, nuestro, etc.)
Mí, notice the diacritic, is a pronombre personal preposicional. For example:
Esta carta es para mí. This letter has been sent to me.
However, yo is also a pronombre personal, but it is a pronombre personal no preposicional. For example:
Creo que yo puedo hacerlo. I think I can do it.
Your second example (Él tiene más libros que mi.) is a common mistake in some Spanish-native speakers (I have noticed it on Peruvian people). As Diego has answered, the correct sentence is Él tiene más libros que yo.
It would be valid if you add something like:
Él tiene más libros que mi hermano. He has more books than my brother.
However, in this case, mi is an adjetivo posesivo.