The reason you can't say what you did is because infinitives in the construction you gave usually have an implicit subject1 and for many speakers, you need subordinate it in order for it to make sense.
As aprendedor mentioned, you need to add on lo with suficiente to nominalize it. But then we get:
Yo hablo lo suficiente para valer la pena ir.
^ explicit YO ^ implicit YO
Because after the para there is nothing to clearly indicate a change of subject, the interpretation is that you speak enough in order to be worth the trouble. As in, you are worth the trouble. Then the ir shows up and it seems to float needlessly.2 Theroetically it could be an infinitival subject, but I can't think of a single Spanish speaker who would interpret it as such.
The easiest fix is to go to the subjunctive:
Yo hablo lo suficiente para que valga la pena ir.
^explicit YO ^ ^ ^aha! We have our subject
| still ambiguous (yo/él/ella/ello/Vd)
alerting potential change in subject
I know you want to use the infinitive form of valer, but because of the way that verb works and what you want to say, there's really no way you can use the infinitive and be easily understood unless you wanted to cheat by using periphrasis, but the same issues apply
Hablo lo suficiente para que vaya a valer la pena ir.
1. Contrary to popular belief, infinitives don't have to be impersonal and can take subjects. They are merely invariable in form (cf. Portuguese which in similar circumstances uses the personal infinitive that inflects only for person but not tense). For example, para beberla él, la bebida tiene que llevar alcohol. Note that some speakers may consider this ungrammatical, but it is described in the RAE's Gramática.
2. This is the same in English, though we construct these clauses differently. Compare I speak enough for it to be worthwhile/worth it to I speak enough to be worthwhile/worth it. When you tack on the extra ir, it's like in English saying I speak enough to be worth it to go.