This is a complex topic in Spanish. In principle the verb must always agree with the noun in number. There are no exceptions of the kind that English shows in, for example, the use of some collective nouns like "family" or "government" or proper names of institutions. So if the subject of your sentence is mi familia then the verb must always be in the singular.
There are a few apparent exceptions to this rule with some noun phrases that include quantifiers such as la mayoría, la mayor parte, la mitad, el 25%. These are singular, but with them you can use either singular or plural in the verb.
La mayoría no quería ir a la playa. ← This is OK.
La mayoría no querían ir a la playa. ← This is also OK, a bit more colloquial.
There are no such exceptions with groups like familia, pareja, matrimonio, trío, cuarteto, etc. There is, though, the possibility of using the plural in subsequent propositions that refer back to one of these groups. That is, it's permissible to speak of mi familia (singular) and immediately refer back to them in the plural:
Mi familia viajó a la playa. Era la primera vez que visitaban el Caribe. Les gustó mucho. Dijeron que quieren volver.
The verb viajó is in the singular, agreeing with familia, but the rest of the verbs (where the tacit subject is mi familia) are all plural, as well as the indirect object pronoun les, because mi familia is semantically plural. This is actually the natural way of expressing this in Spanish. If you used all verbs in the singular, even though it would be strictly speaking correct, it would sound extremely awkward.
There is also an exception, only found in colloquial speech, when the collective noun includes the speaker; in this case a singular collective noun phrase can go with a plural verb in the first person plural. There are not many possible examples for this though:
Toda la familia iremos a la playa.
For reference: the Nueva Gramática de la Lengua Española (NGLE) deals with this on section 12.4k and around.