Both are perfectly fine. The trick is that the ver verb has a lot of meanings, and depending on the sentence its meaning can be different. If you just say:
La chica que vi ayer se llama Ana.
Here I would say that you just saw the girl as you were walking down the street of whatever. Another interpretation could be that you are a doctor and the girl came to you so that you could check her. But if you say:
La chica a la que vi ayer se llama Ana.
Now the ver verb could be interpreted as meet: you saw the girl because you had an appointment to go see her. But this is indeed quite prone to interpretations and the final meaning of the sentences will depend on the context.