Weird Issue with SSH Keys

I generated ssh keys (public and private) and stored them in ~/.ssh/myServer/. I added the public key to the server's ~/.ssh/authorized_keys file and everything works great! In my effort to try and understand how the keys work I deleted them and was surprised to find that I was still able to connect to the server. Why is it that I am still able to connect without a password?