Are there sources that any minute not learned Torah is a sin?

I was recently referred to the Even HaEzel on the Rambam in the 3rd chapter of Hilchos Melachim. Reb Isser Zalman Meltzer (the author) writes there that there is no sin of Bittul Torah for a regular individual (i.e. not a king) if he is doing something to enjoy himself. This would imply that reading a novel, shmoozing, or anything one does for his enjoyment (as long as it's not a sin, obviously) is not considered Bittul Torah.

Are there any Rishonim or Achronim which disagree with this and hold that every minute you can learn and don't is considered Bittul Torah? I would also appreciate any other sources which hold like the Even HaEzel.