Delete key doesn't work in vim on iTerm2

Firstly, I typed some thing on insert mode, like

Hello

At this stage, if I didn't exit from insert mode, delete works and can delete the whole word if I want.

If I quit insert mode, and open insert mode again, this Hello can't be removed by delete. But the newly typed in content can be removed.

I've renamed my .vimrc, problem still exits.

And I did some test on remote server through iterm, didn't encounter the same issue.

What could be the cause of this problem? How to fix it?