If you are saying your mother
it will be very rare that you would use "su madre" because "su" is for the 2nd person "Usted", and that puts the sentence in a very rare context, talking to someone with a lot of respect and talking about his/her mother, not saying it's impossible but it will be very rare. If you fall in ambiguity you can add what's necessary to remove that, but I think the context could help, maybe while you're talking to the person who's mother is looking for the sister of the other people, you could point or make some gesture that can help you, remember that the bodt language is also very useful, anyway it could be like this:
Your mother is looking for your sister.
Tu madre busca a tu hermana.
Su madre busca a su hermana.
Your mother is looking for their sister.
Tu madre busca a su hermana.
Su madre busca a su hermana.
Vuestra madre busca a su hermana.(old-fashioned)
Tu madre busca a la hermana de ellos/ellas.
Su madre busca a la hermana de ellos/ellas.