Both Spanish forms are correct, but have slightly different senses of meaning--slight enough that it's hardly worth pointing out the difference, and some will say they're the same. Still, the conditional clause for one is subjunctive, while the other is conditional.
There is no direct correlation to English for these two forms, so consider my "back to English" translations as suggestive for the actual meanings, but it may help one to see that the sentences, while both being acceptable and very similar, do differ, albeit the difference is insignificant.
SP: Si ellos hubieran puesto atención en la clase de cálculo, hubieran entendido el examen mejor.
EN: If they had paid attention in Calculus class, they should have better understood the exam.
SP: Si ellos hubieran puesto atención en la clase de cálculo, habrían entendido el examen mejor.
EN: If they had paid attention in Calculus class, they would have better understood the exam.
NOTE: This "should have" usage in English is becoming outmoded. It does not, in this context, refer to obligation or duty, but is rather a formal expression of the conditional mood. It is quite correct, but is less common in current idiomatic English. This may align with the Spanish usage as well. The most common form would be the past subjunctive + the conditional, as in the second example, though both forms are acceptable. The subjunctive verb in the conditional clause in Spanish may also be considered more formal, albeit less common.