Gemara

(Beraisa): One who did not engage in Torah cannot say (to the Beis Din of Shamayim) that his wealth or poverty made it impossible to learn.

If an Oni says that he was too poor, they tell him 'You were no poorer than Hillel. He used to earn one coin each day. He gave half to the Beis Medrash attendant, and the other half he used to support his household...'

If an Ashir says that he was too distracted with his wealth, they tell him 'You were no richer than R. Eliezer ben Charsom! His father left to him 1000 cities and 1000 ships. Each day he would take a flask of flour and go from city to city to learn Torah.'

If a Rasha says that he was distracted by his Yetzer ha'Ra due to his beauty, they tell him 'You were not more beautiful than Yosef. Every day Potifar's wife tried to seduce him. She wore different clothes for him every morning and evening.'

Eruvin 54a: One should engage in Torah even if he has pain in any place:

If his head hurts, he should learn - "Ki Livyas Chen Hem l'Roshecha". The same applies to a pain in his throat ("...va'Anakim l'Gargerosecha") stomach ("Rif'us Tehi l'Sarecha"), bones ("v'Shikuy l'Atzmosecha") or his entire body ("ul'Chol Besaro Marpei").

Berachos 5a - R. Yakov bar Idi or R. Acha bar R. Chanina: Afflictions of love are those that do not cause Bitul Torah -- "Ashrei ha'Gever Asher Teyasrenu Kah umi'Torascha Selamdenu";

The other of R. Yakov and R. Acha: They are afflictions that do not cause Bitul Tefilah -- "Baruch Elokim Asher Lo Hesir Tefilasi v'Chasdo me'Iti".

R. Aba brei d'R. Chiya bar Aba: Both of these are afflictions of love - "Ki Es Asher Ye'ehav Hash-m Yochi'ach".

Question: But it says "umi'Torascha Selamdenu" (Hash-m will teach him, i.e. he will still be able to learn)!

Answer: We read this Selamdeinu (the Torah teaches us that one who is afflicted is fortunate. Maharshal - we read it Limadtanu, You taught this to us.)

Avos 4:9: An Ashir who is Batel from Torah will eventually become poor. One who fulfills the Torah amidst poverty will eventually become rich.

Rishonim

Rambam (Hilchos Talmud Torah 1:8): Every Yisrael is obligated to learn Torah, whether he is poor or rich, healthy or afflicted, young or old and weak.

Source (Perush ha'Mishnayos Avos 5:22): Siv u'Vale Bah - even in Seivah (old age) do not veer from Torah.

Rambam (ibid.): Even an Oni who goes begging, even one with a wife and children must fix times for Torah by day and at night - "V'Hagisa Bo Yomam va'Laylah".

Source (Chidushei R. Meir Simchah Sotah 42a DH Amar): This is from Tanchuma Beshalach (20).

Poskim

Shulchan Aruch (OC 156:1): After learning one should attend to his business. One's business should be haphazard and his Torah should be constant, then both of them will succeed.

Chachmas Shlomo (DH veha'Gaon): The Hafla'ah explains that one who is distracted with business must Kove'a i.e. 'steal' time for Torah.

Mishnah Berurah (2): Each day one should work as much as is necessary. The Yetzer ha'Ra entices people to think that they must work the entire day. Rather, one should imagine what he would consider sufficient if he had to feed and clothe another person, and do this much for himself (Sha'ar ha'Tziyon 3).

Shulchan Aruch (YD 246:25,26): If one could engage in Torah and does not, this is "Ki Devar Hash-m Bozah". An Ashir who is Batel from Torah will eventually become poor; one who fulfills the Torah amidst poverty will eventually become rich.

Question: Some people fulfill Torah amidst poverty and never become rich, e.g. R. Elazar ben Pedas (Ta'anis 25a)!

Answer #1 (Tosfos Yom Tov Avos 4:9 DH Sofo): The Mishnah discusses someone of an average Mazel.

Answer #2 (Tosfos Yom Tov, ibid.): When Hash-m sees that wealth would distract a person from Torah he withholds wealth from him.

Answer #3 (Tif'eres Yisrael Avos 4:9:53): There are sins that can nullify the 'promise'. We find that Yakov was concerned lest sins nullified Hash-m's promises to him. Also, perhaps the person learned Lo Lishmah.

Note: Tif'eres Yisrael discusses people who appear to be Tzadikim. Regarding R. Elazar, Chas v'Shalom to say that an Amora, especially 'the master of Eretz Yisrael (Gitin 19b)', learned Lo Lishmah.