The most idiomatic way to say it is
Me quedé triste porque/cuando te fuiste
which does not require subjunctive because it uses porque instead of que.
If you want to use que, you could also say
Me quedé triste de que te fueras
which is grammatically correct but is less common (0 results on Google!). This document is a good reference for when to use subjunctive. See on the second page, triste + (de) que requires subjunctive. Note that in this case, you would need de to introduce the relative clause. See this answer for details.