only thing i could possibly think of is that it's possible VLC demands higher GPU clocks simply being open, and it's possible that your game wasn't doing that... if you want to test that, there's a program called ClockBlocker (lol) that does the same thing. it forces max clocks when certain programs are open. it may need a little tweaking and experimentation and be sure to read the documentation/UI item hover tips for it
if you do this i'd be interested in knowing if it works. i use clockblocker for my RX 570 because it doesn't force high clocks when certain games are open (source games specifically). some say this is because it doesn't need those extra cycles at all, but there's some evidence against that
another thing you could try out: disable hardware acceleration in VLC and see if it still works. see if it has anything to do with that
