Why did they say "Lo Es Ba Es Beis Hashem Lehibanos"?
Rashi: Their enemies stopped them from building it. At first, they thought that I will not fulfill My word "l'Mal'os l'Charvos Yerushalayim Shiv'im Shanah" (9:2), and the time to build it will not come again 1 . He told them, now the time came. The first time [when they started building with Koresh's permission] was 70 years from the beginning of Malchus Bavel. Really, it depends on 70 years from Churban Yerushalayim in the days of Tzidkiyah, 18 years after Malchus Bavel. They were exiled in the 18th year [from the conquest of Yehoyakim], and [it was] the 19th year [of Nevuchadnetzar's reign] (Megilah 11b). The 18 extra years finished now.
Malbim: The nation had a tradition that before the true Ge'ulah, there must be walls in Shamayim and Eretz, and the war of Gog and Magog. All who prophesized about it said so. The people did not see any of these preparations. Therefore, they said that it is not time yet. Also, a stipulation of the Binyan is that it will be rest from their enemies, like it was at the time of Shlomo's Binyan. This was not a time of rest - they were subjugated to the kings of Madai and Paras! The time of Ge'ulah must precede the Binyan!
Radak: They despaired, even though they knew that the 70 years from the Churban finished now. They should have realized that Hashem did not raise them from Galus without reason! They should have aroused themselves even without the Navi telling them. They saw that their deeds were not succeeding - "Zeratem Harbe v'Havei Me'at" (6). They should have realized that this is their punishment for despairing from building the Beis ha'Mikdash! They did not arouse themselves, so Hashem sent to them via Chagai.