Quick Time screencast slows down my gameplay also size too big

I recorded few of the videos in past with quicktime. Size of those videos is too big (~5GB). I have a macbook pro 15-inch retina version. For cutting down the size I am using HandBrake. With that a 2 GB video would come up to 45MB but then video resolution goes really down. Is there any other recording software that can record videos at relatively smaller size.

I have another issue, when I record a gaming video with quicktime the game gets really laggy and impossible to play fluently. I want a screen recorder which doesn't take much resources and my laptops gaming performance doesn't decrease.