i know this is for classic and my answer for classic would be that as a team of 3 you would only miss out on the convenience of instant solo 5man dungeons. As 3 you will likely have an easier time raiding for gear and the difference between the two teams would be negligible.

As for retail, MoP was really trying hard to insert the 3man meta events, but people hated it, BFA tried again, but people hate it. Arenas were 3man, but they handicapped us there. I dropped to 3 during MoP, and i think it was the only worthwhile time to do so. BFA i considered it until the /follow changes to arena. I think they've given up on the idea of 3man events, because Shadowlands is extremely 5man friendly. All the stuff i'd expect to be 3man scenarios are set to 5man in Shadowlands. Especially one of their main sellers, the tower and the corruption scenarios. The corruption scenarios are looking to be some kind of cross between Garrisons and Withered army training, but doable in 5mans.

So for classic, 3man teams are great if you raid for your gear and dont mind finding 2 others for 5mans.
For retail, 5man or more is the only way to go imo.