I think what you say is largely true, and it's pretty much the rationalization I used when I was a premie. I thought he was a bad speaker, without much charisma, but I told myself this was a test for me, and that it didn't matter because he was who he was and that's all that meant anything.
But I think there are also premies who believe Rawat is a highly inspirational speaker and very charismatic. These are the ones who are let down when they bring people to hear Rawat and their guests are polite, probably, and just never mention it again. They are not impressed, nor inspired. The cult rationale, is that these people lack "thirst."