One Contact Receiving My Messages Under An Email, Not My Contact Name

My iMessage is working with all iPhone users, except with a specific contact. iMessage is working with this contact, except that he is only able to receive my iMessages as my email address and not from my phone number. I use my email as my ID for both iPad and iPhone. I have my send & receive settings set to only my phone number, so when i try to send an iMessage to this contact, it automatically fails to deliver. Anything i can do about it? Is it something to do with his settings or mine?