To speed things up, I'd look into whispercpp, it should be faster without any quality loss.
With that said, I've never used it so I don't know for sure and I've always found comparing quality to be very difficult with whisper since it never gives the same result twice, even with the same settings.