Neither, actually.
There's no need for a reflexive pronoun if you're going to use ser or estar. You could use the se pasivo in place of the true pasive (that uses ser), however. The following to sentences are, absent any agent, roughly equivalent:
- Habían sido demolidas aquellas casas (… por alguien/algo). The houses had been demolished (... by someone/something).
- Se habían demolido aquellas casas. The houses had gotten demolished.
Notice the agent can only be added in the first. If you use estar, however, we are no longer talking about the act of demolishing, rather the result of their demolition.
- Habían estado demolidas aquellas casas (desde …). The houses had been demolished / in a demolished state (since …).
In this case, you can substitute quedar or resultar for estar and it might make it a bit clearer what's being said: había quedado/resultado demolidas aquellas casas.
Se demolieron says that they got demolished, but doesn't allow you to specify who demolished them. Also, it happens contemporaneously in a past time frame. Since using the pluperfect has it happen before a given time frame, context will dictate whether you need to use one or the other.
So ultimately, to decide which form to use, you need to think whether you want to focus on the action (use true passive or se pasivo) or the result (use estar), whether you want it as part of the past narration (use preterite) or something that happens before the narration (use pluperfect or even antepreterite), and if you are focusing on the action, whether you wish to admit an agent (use true passive) or if that's not important (use se pasivo).
In general, Spanish speakers don't like stacking verbs as much as we do in English1, and so prefer se pasivo over true passive. However, I have noticed my friends from Central America using true passive far more often than speakers elsewhere, but that could also potentially be due to influence from English as they're all native bilinguals.
1. Compare He couldn't've been going to see the movie which sounds quite natural in English to its translation No podía haber estado yendo a ver la peli which sounds positively wretched, although perfectly grammatical.