How can I set up seamless navigation of vim and iterm2 splits?

I am trying to map my C-h, C-j, C-k, C-l keys to be able to navigate vim (running in iterm2) splits but if there is no vim split to move to it should figure out that I want to jump to a different iterm2 window.

Pretty much this http://ift.tt/1keaRxD but without having to run tmux (or at least without needing tmux to do the multiplexing)