How can someone become good at playing mind games with their opponent in fighting games?
By practicing it deliberately.
Specifically you should watch your opponent’s patterns. What do they keep doing and how can you exploit that? Watch what they do in each situation, get a feeling for their tempo and reaction time. If their reaction time is better than yours, then you need to beat them by acting on the tempo. If they do not adhere to the tempo, then you need to figure out by how much, and either act first to interrupt their options, or act second to punish them.
Watch for common player behaviors and keep a mental record of those. One example of this is, as Marth, I like to run through my opponent, then run cancel with a crouch, and fsmash back at the opponent I passed by. This is because when you run through, many people think they’re safe and do an option out of shield. However this is not foolproof. Players with good reaction time can grab me out of shield before I run through them. Players who are smart can recognize my pattern and either jump out of shield earlier, or hold onto their shield so my fsmash does nothing. At which point the correct response from me is to notice they are doing this and instead do run through, cactaur dash (run cancel and dash opposite direction), grab, because I’ve conditioned them to stay in shield.
Think about how everything you do conditions a response from your opponent and other things you can do instead that beat that response. If you do something that is exploitable, change it up in expectation of your opponent catching on. Watch what you do before you do an action, because that might give it away. Similarly watch for that in your opponent.
Getting good at mindgames is about studying other people, and finding 50/50 scenarios.
Also read this guide.
Here’s 3 other guides on it as it applies specifically to smash bros (though you can extend these lessons outside of those games too)
And here’s a paper on people’s patterns in Rock Paper Scissors and a basic guide to winning:
(The short is, winners tend to stick with their choice more often, losers tend to switch more often, and continue switching to unused options.)
Think about what the opponent is actually doing. Remember their responses to scenarios, and keep updating to do the thing that will beat their current pattern. If you have found a pattern that keeps winning, keep doing it, or if it’s just a pure mixup, switch after 2-3 reps, because that’s when your opponent is likely to switch, unless they’re bad and don’t understand the counterplay of the different options.
Of course also look for scenarios in which you can cover all or most of your opponent’s options on reaction and just setplay them. Then you don’t need to guess.
The beauty of competitive games is that there’s a complicated web of counters to different options in different scenarios, with one covering many in many cases, and different ones changing in utility based on the scenario. But to exploit these, you really need to think and pay attention, or you’ll get played.