When I watched this as a toddler I got quite scared by the Ghost that appears twice in the Special plus the Painting and Skeleton can be a bit frightening. If your kid isn't scared easy then yes show them it.
I didn't like much of the DVD except for Julia Roberts scene and the Frazzle song. Plus, unlike most kids, the last scene spooked me because Elmo didn't say goodbye and ended with that C-R-E-E-P-Y portrait!!! I say, use this album, not the home video. Okay?