Have been using it casually but now I wanted a quick way to follow internal links in my Thesis and go back.

Zathura can do this and not just this apparently!

zathura(1) — Arch manual pages

• Tab goes to index mode, where space/enter follow the link in the index
• f shows links that can be followed by typing the number and then enter
• ^o, ^i: Move backward and forward through the jump list! Practically ^o is basically “go back”.

It even has a config file, with remapping, design and stuff zathurarc(5) — Arch manual pages

Downsides:

• the numbers shown w/ f are too small, and no way to change them
• the follow-on-single-click option is in newer version than what I have access to (I should reinstall my system actually)