Added memory does nothing to performance.

IMac, 27" screen. Came with 4 gig of memory. I never used all 4 gig until I started using IMovie. Bought 8 gig extra memory, now have total of 12 gig. Nothing speed-ed up at all. When monitoring the memory, it seems that I'm always using over 8 gig and can only clean up the memory to about 6 gig. How did I ever get by with 4 gig? Nothing adds up.