This is an ambiguous situation. At first it looks undoubtedly like a mistake. I wondered first whether this could be excused because sometimes common idioms become fixed to the point of ignoring basic grammatical rules when used in informal contexts. But here the grammar is actually inconsistent within the context; me han valido is plural, which shows the speaker respects number agreement, but me valió is singular, which shows lack of agreement. So this could just be a mistake.
But there's another possibility. What is the subject of the expression valerle a uno un carajo algo? The structure is
Subj valer reflexive-IObj DObj[un carajo]
where IObj = indirect object, DObj = direct object, the expression un carajo, and Subj = subject (movable). This subject can be explicit:
- Tus quejas me valen un carajo.
- Me vale un carajo lo que digas.
It can also be implicit: just saying Me vale un carajo in a given situation signals that you don't care about it. There's no real referent to that implicit subject; it's the context as a whole. The context can include a time, a place, the company of certain people, etc.
What I'm trying to arrive at is that an expression like las veces que... can be such a context. In Spanish, there's a certain set of nouns that can form complements of time by themselves, without resorting to any preposition or any other overt indication:
- El otro día vi a María en el mercado.
- La vez pasada no quisiste atenderme.
- Nos enamoramos la primera vez que te vi.
- Juan va de compras los lunes.
What I'm proposing is that las veces, in your text, is one such complement of time, and that the implied subject of me valió in that last sentence could refer not to las veces but to the context as a whole. In that case the sentence could be reworded more "properly" as
... fueron las veces en que la situación me valió un carajo las que hicieron toda la diferencia.
This is difficult to translate into English, but the subject of whatever you translate as me valió... would be a dummy it.