Teaching is harder than hell because dealing with dozens of kids during the course of a day or a class of 20-30 or more a day is difficult.
Little of the job can be delegated, and you must be "on" all the time. There is no time to mentally "remove" yourself from your work, unlike business or other public sector jobs. Then, especially in the elementary grades, is the specter of always getting sick. Eventually that takes a toll on a teacher's body.
You have to not only be familiar at least with your subject matter, but it is vital to know which methods of teaching work best with different students. In special education classes, it's like a three-ring circus and is difficult to manage. You have to know HOW to teach, which is more important than WHAT to teach. You have to understand child development to be able to do the job.
The hardest and worst part of the job is putting up with idiot or insane administrators. You have utterly no control over your job.