Sounds like the worst idea ever but for our family, it worked. Not only did it work, but it was fun!
One thing I love about Disney World is that no matter what your situation, you are all in it together. No one expects perfection and all the parents are there encouraging and helping each other out! You feel like part of the Disney family. So remember that when your child is throwing a Mickey shaped pretzel in your face because you looked at them wrong.
When conquering Disney World with children, you need to come prepared but also ready to go with the flow. The biggest tip I can give to you is to make reservations and use your Fast Passes. The worst thing you can do is show up to a restaurant, exhausted from your day, and have to wait an hour and a half because they are booked. And they will be. All the best dining options are usually booked up 180 days out. This is the moment a good Disney Specialist (*cough cough) would come in handy. They can help you book those reservations and even tell you what places works best for your family. It also benefits EVERYONE to have your Fast Passes booked so you aren't waiting in long lines. I always pack a bag with snacks and games for my kids but after 45 minutes I even start to lose interest. If you get the privilege to stay on property at Disney World you can book your Fast Passes 60 days out. If you are staying off, you can book 30 days out. It doesn't sound like a huge difference but it is. Disney only opens up a certain amount of Fast Passes per ride, per day, so you will want to grab them while they are hot.
Lastly, you need to be prepared to go with the flow. Sometimes you won't be able to make your reservation in time, sometimes you will miss your Fast Pass and your kid may all of sudden hate everything Toy Story related (after loving it for years) the moment you are about to get on Slinky Dog. Take a deep breath and give yourself a break. Your kids won't remember these hiccups. They will remember the first time they see the castle. They will remember the rides. They will remember spending time with their parents who can still act like kids.