The Gem. (Ber. 33b) says that one distances himself from a ''Shor'' (and is also Mafsik in Tefilla) since in the month of Nissan, it feels a certain freshness and is stimulated to excitable behavior and is therefore dangerous.
Rashi adds, for example, it sees the greenery appearing and its mind becomes unsettled, and the Yetzer Hora enters it.
Now, on 61a, the Gem. says that since the Shor bites and kicks, we see indeed that it possesses a Yetzer Hora.
Why is this an indication that it has a Yetzer Hora? Excitable behavior is an instinct in an animal, which it has little or no control over. How is this associated with the Yetzer Hora? A man also is capable of animalistic behavior (towards which his Yetzer Hora may push him) but man has an intellect which directs him how to behave and hopefully overcome the Yetzer Hora's suggestions. An animal lacks these powers of advanced reasoning
The Yetzer Harah is none other than that animalistic instinct, which, as you have pointed out, the person can control and the animal cannot. In the animal it relates to the urge to be destructive.