The Talmud states in 2 places (Menachos 100a and somewhere in the sixth chapter of Yoma) that the "Bavli'im" were hated and so were labelled as the people who pulled the hair of the scapegoat and ate raw sacrificial meat, even though they actually had nothing to do with these things. Tosfot in Menachot explain that the origin of this hatred is in Yoma 9, where Raish Lakish tells an Amora from Bavel (in interpreting a pasuk of shir hashirim) that the reluctance of the Bavli'im to ascend to Israel inhibited the schinah from dwelling in the Second Temple.
My question is as follows: The culmination of the sugya in Yoma 9 is that Rabbi Yochanan told Reish Lakish that the reason for the absence of the schina in bayis sheini was not because of the Bavli'im but rather because the schina can only rest in a Sanctuary built under the auspices of bnei Shem (and not bnei Yefet)! Thus, there is no reason to hate the bavli'im!
Rebbi Yochanan certainly holds as you mention, that there is no reason to reject the Bavli'im, and that it was not because of them that the Shechinah did not dwell in the second Beis ha'Mikdash. However, Reish Lakish there argues with Rebbi Yochanan. Even if we propose that Reish Lakish's silence to Rebbi Yochanan's statement indicates that he agreed with him, nevertheless the rest of the people of Eretz Yisrael (who referred to anyone they hated as "Bavli'im") upheld the original view of Reish Lakish and blamed the Bavli'im.
It is worth noting that Rebbi Yochanan is following his opinion as expressed elsewhere. In Shabbos (145b), when the students in the Beis Midrash questioned why are birds in Bavel fatter, their festivals happier, and their Talmidei Chachamim stand out, Rav Ashi answered for all of them and said that they are results of the inferior quality of the people of Bavel. Rebbi Yochanan disagreed and attempted to defend the people of Bavel, and he answered the questions in a different manner (see there). (We also find this in Yevamos 63b, when Rebbi Yochanan empathized with the pain of the people of Bavel.) Perhaps the reason for this is that Rebbi Yochanan, who authored the Talmid Yerushalmi, saw that the continuity of the Oral tradition and Halachic process would come from Bavel and the Talmid Bavli (due to the increasing harshness of the decrees of the Roman persecutors in Eretz Yisrael at that time) -- see RIF in end of Eruvin -- and therefore he wanted to build up the reputation of the people of Bavel in the eyes of the people of Eretz Yisrael. We find (in Sanhedrin 24a), similarly, that Rebbi Yochanan praised the Talmid Bavli.
Mordecai