Rate Limit was designed to defend against such attacks among other numerous applications. Before we continue, go ahead and enabled Rate Limit from WHM -> Varnish -> Rate Limit). Once enable it, you can then begin setting rules to rate limit URL accesses. Let's walk through the wp-login.php example. By default your Rate Limit page will have this:
wp-login.php 3req/s 10req/30s 30req/5m
The first bit of information is the page or URI (/wp-login.php is a URI). The next three bits of information represents the three rate of accesses which if exceeded will lead to a block from Varnish with HTTP code 429 (not firewall block). So this will effectively prevent further brute force attempts. So if you were to reach wp-login.php more than 3 times per second OR 10 times over 30 seconds OR 30 times over 5 minutes, you get blocked. When either of the three limits is reached, the attacker is blocked by Varnish.
whoever (or whatever) attempts to attack wp-login.php will receive this message on their browser:
Error 429 "Slow down!"