Know that by running Varnish, you are essentially saving a lot of memory. The caching aspect of Varnish makes it that the X amount of memory you allocate to it is shared amongst several visitors. For example, a PHP script that takes 16MB of memory to run for one connection will now take 32KB of memory when running with Varnish. Do the math and see how much performance and capacity you're squeezing out of your current hardware!
When Varnish can't find additional memory to cache objects, it leverages the Linux kernel to move unused objects to the disk. Varnish is so efficient that visitors barely notice any of the cache management business. It's all transparent and fast.