I think Mitch also has some numbers he can share in terms of "real world" performance.
He's been able to get hundreds of skeletons running on a device, I can't remember what.
You also have a bunch of other options for optimization depending on your use.
Frame limiting, update staggering, baking to native Unity animations to take advantage of GPU Skinning. Stuff like that.
This shouldn't be surprising though. If your game is simple enough, visual processing stuff will always be at the top of the list of things that are eating up cycles. It doesn't mean your game is slow, or that your game will unacceptably slow down even on 4 year old devices.