IIS will respect les deux of those limits.
If you set a 500MB private byte limit, as soon as a worker process attempts to commit 501MB, IIS will spin up a nouveau worker process et kill le old one.
If you set a 3GB virtual memory limit, as soon as a worker process attempts to reserve 3.001GB, IIS will spin up a nouveau worker process et kill le old one.
If you are on a 64bit platform, you should be aware that ASP.NET applications aggressively reserve virtual memory. As an example, J'ai an app on a farm that uses seulement 88MB of private bytes, mais its sitting at 5.4GB Virtual Size right now. Je crois le virtual memory reservation is a function of physical RAM on le server. C'est aussi important to understand that on a 64bit platform, reserving large portions of virtual memory has zero performance impact.
Fondamentalement, si you are having memory consumption issues on an IIS server, le setting you want to limit is Private Memory/Bytes, this is what corresponds to actual memory usage.