REFERENCES ADDED TO SUPPORT MY ANSWER BELOW:
REFERENCE 1:
The usage described in the question is confirmed by this thesis I found on the Internet (page 95 of the pdf, page 87 of the paper), which states that the verb "ir" in constructions followed by "a" + infinitive can be interpreted as a periphrastic form (going to do sth) or as an occurrence of the verb "ir" with its full lexical force (where "ir" does not express willingness or proximity of the action but actual movement):
Se advierte que la construcción ‘iban a trabajar’ se puede tomar como una construcción perifrástica o como la aparición plena del verbo
ir:
(a) Iban a trabajar: se desplazaban de un lugar hacia su trabajo. [The bolds are mine.]
(b) Iban a trabajar: se disponían a trabajar, comienzo de la acción trabajar.
REFERENCE 2:
Here is another author who confirms the above: the verb phrase "ir a" + infinitive can have either of the two meanings mentioned above, modal or locative (see item 2.1 on page 2):
En (5) se dan dos lecturas de ir a + INF: una léxica, de desplazamiento físico, y otra intencional, en cuyo caso el hablante conceptualiza la acción de comprar como un proyecto que, desde el
punto de vista temporal, se sitúa en el futuro:
(5) Voy a comprarme un libro.
(What the author says is that "ir a" + infinitive can have a lexical reading where there is movement or a modal reading expressing future intention.)
REFERENCE 3:
In this site the following can be read on page 1 of the pdf document:
Como nota el Manual, la aparición de ir en pretérito perfecto con el
significado de (20) está limitada a los verbos télicos, que expresan
procesos delimitados, por lo que se excluyen estados (21.a–b), pero
también actividades, con las que ir solo puede recibir una
interpretación como verbo léxico de movimiento (21.c):
(21) a. *Fueron a ser felices.
b. *Fue a haber una catástrofe.
c. #Fueron a trabajar / caminar.
(What the author says above is that, with "ir" in the past tense and an action verb like "trabajar" or "caminar", "ir" can ONLY be interpreted as a verb of movement.)
MY ANSWER BEFORE ADDING THE REFERENCES ABOVE:
In English, "work" can be a noun or an infinitive in "I'm going to work" depending on whether you refer to the place where you work or to the action: Voy al trabajo / Voy a trabajar.
Now, in Spanish the infinitive can indeed be used to refer indirectly to the place where you do an action. Proof of this is that, in answer to the question ¿Adónde vas? or ¿Dónde vas?, we can say Voy a + infinitivo. We can understand that, in these cases, we are eliding "to the place where I ...". I'm going to provide a few examples below (I'm using questions with "you" and answers with "I", but other persons can of course be used). Please note that in all of the cases, and mainly in (4), we can use the pronominal form "Me voy a ...":
(1)
A. ¿Adónde vas?
B. Voy a trabajar. (Literally, I'm going to work, meaning: I'm going to the place where I work.)
(2)
A. ¿Adónde vas?
B. Voy a estudiar. (Literally, I'm going to study, meaning: I'm going to the place where I study.)
(3)
A. ¿Adónde vas tan bien vestido?
B. Voy a comer. (Literally, I'm going to eat, meaning: I'm going to a place to have a meal, i.e. I'm going to eat out.)
(4)
A. ¿Adónde vas?
B. Voy a dormir. (Literally, I'm going to sleep, meaning: I'm going to a place to sleep, i.e. I'm going to bed.)
(5)
A. ¿Adónde vas?
B. Voy a comprar. (Literally, I'm going to buy, meaning: I'm going to a place to buy, i.e. I'm going shopping.)
(6)
A. ¿Adónde vas? (B forms part of a choir)
B. Voy a cantar. (Literally, I'm going to sing, meaning: I'm going to a place to sing, i.e. I'm going to choir practice.)
(7)
A. ¿Adónde vas?
B. Voy a pasear. (Literally, I'm going to walk/drive, meaning: I'm going for a walk/drive.)
(8)
A. ¿Adónde vas?
B. Voy a bailar. (Literally, I'm going to dance, meaning: I'm going to a place to dance, i.e. I'm going to a disco.)
You can of course also use a noun to refer to the place or to the activity. Therefore, in answer to the questions above, (B) can reply:
(1) Voy al trabajo / Voy a la oficina.
(2) Voy a clase / Voy a la escuela/universidad.
(3) Voy a un restaurante.
(4) Me voy a la cama. (Here we tend to always use the pronominal form "irse", which is optional in the other cases.)
(5) Voy de compras / Voy al supermercado.
(6) Voy a canto / coro.
(7) Voy de paseo.
(8) Voy a la disco.
Just as in the English "go to work", in Spanish the phrase "ir a + infinitive" can mean "go where you do an action" or "go and do an action". It all depends on the context. In the examples I provided at the beginning, since the question was Where?, it is clear that the infinitive refers to the place where you do the action, regardless of whether you end up doing the intended action or not.
What I explained above also applies to other verbs, like "llevar", "traer", "acompañar", "venir":
(9)
A. ¿Adónde llevas a María?
B. La llevo a trabajar. (= La llevo al trabajo.)
(10)
A. ¿Adónde traes a Pedro?
B. Lo traigo a estudiar. (= Lo traigo a la escuela.)
(11)
A. ¿Adónde acompañas a Juan?
B. Lo acompaño a comprar. (= Lo acompaño a la tienda.)
(12)
A. ¿De dónde vienes?
B. Vengo de jugar. (= Vengo de la cancha / del club.)
It is true that in some cases the infinitive with "a" may convey some purpose meaning apart from indirectly indicating the place where the action is performed.
Note: I don't agree with the other answer that this is a case of verb nominalization. That would be the case if the infinitive were used as a subject or as an object, which are typical nominal functions. In English, we typically use V-ing in these cases, but the infinitive is also possible:
Though less idiomatic, when it works as a noun the infinitive can take the article "el". Sometimes, to work with "el" the infinitive needs some complement:
The article is not allowed where the infinitive is only that: an infinitive.
We CANNOT say: *Voy al trabajar.
As I said in the second paragraph after the references I added at the beginning to support my answer, the infinitive can be used to refer to the place but is NOT the place. In the phrases Voy a trabajar / Voy a estudiar, etc. referring to going to the place where you do those activities, there is simply an omission of "el lugar donde" (the place where I ...)
With the verb "ir", there are two possible interpretations: (1) "ir" indicates movement and the infinitive indirectly refers to the place, or (2) "ir a" indicates intention to do the action indicated by the infinitive.
With transitive verbs like "llevar" or "traer", there are also two possible interpretations: (1) the infinitive indirectly refers to the place; (2) the infinitive refers to the action done or to be done by the object alone or together with the subject:
(1) Como no puede dejar a su bebé con nadie, lo lleva/trae a trabajar (lo lleva/trae al trabajo). (She takes/brings the baby with her to work.)
(2) Siempre lleva/trae a José a trabajar. (She always takes/brings Jose to work alone or with her.)