• Showcase

  • Spine Pro Vtuber Working Prototype

    Englisch

  • Misaki

  • Beitrag Nr. 115
  • Dieser Beitrag ist in Englisch

The expressions feature is really fun and wonderful!! 😃 I haven't been able to test it much because preparing the animations still takes more time, but here are the results of a quick test I did:

I know that the transition animations back to the default pause also need to be registered in the expression buttons, but I have not yet been able to do that at this time.

My current model is in a very half-assed state, but I'll leave the data here for anyone who wants to test it:
chara-for-Spine-Vtuber-Prototype_20230130.zip
When I make more improvements, I will share the data here again.

Related Discussions

I should add another setting for 'returning to default pose' animation to the customizable expression feature. That way you save another expression slot.

After this feature, facial expression recognition AI should be as easy as setting which expression slot you want to use for the AI result.

2 Monate später

Spine Vtuber Prototype body track test 1. Preview only.

    • Misaki

    • Beitrag Nr. 118
    • Dieser Beitrag ist in Englisch

    SilverStraw Wow, the tool can finally capture body movements!! That is definitely an inovative update 😃 I'm really looking forward to the day when it will be available!!

      • Erika

      • Beitrag Nr. 119
      • Dieser Beitrag ist in Englisch

      This is amazing! looking forward to creating a cute prototype for this when I have time too >.< thank you for your work!

      2 Monate später

      There is an updated tracking framework but I would have break down the application. It is almost like starting over. Wish me luck 🙏.

      Some possible gestures if it works 😰.

      Face gestures
      1 - browDownLeft
      2 - browDownRight
      3 - browInnerUp
      4 - browOuterUpLeft
      5 - browOuterUpRight
      6 - cheekPuff
      7 - cheekSquintLeft
      8 - cheekSquintRight
      9 - eyeBlinkLeft
      10 - eyeBlinkRight
      11 - eyeLookDownLeft
      12 - eyeLookDownRight
      13 - eyeLookInLeft
      14 - eyeLookInRight
      15 - eyeLookOutLeft
      16 - eyeLookOutRight
      17 - eyeLookUpLeft
      18 - eyeLookUpRight
      19 - eyeSquintLeft
      20 - eyeSquintRight
      21 - eyeWideLeft
      22 - eyeWideRight
      23 - jawForward
      24 - jawLeft
      25 - jawOpen
      26 - jawRight
      27 - mouthClose
      28 - mouthDimpleLeft
      29 - mouthDimpleRight
      30 - mouthFrownLeft
      31 - mouthFrownRight
      32 - mouthFunnel
      33 - mouthLeft
      34 - mouthLowerDownLeft
      35 - mouthLowerDownRight
      36 - mouthPressLeft
      37 - mouthPressRight
      38 - mouthPucker
      39 - mouthRight
      40 - mouthRollLower
      41 - mouthRollUpper
      42 - mouthShrugLower
      43 - mouthShrugUpper
      44 - mouthSmileLeft
      45 - mouthSmileRight
      46 - mouthStretchLeft
      47 - mouthStretchRight
      48 - mouthUpperUpLeft
      49 - mouthUpperUpRight
      50 - noseSneerLeft
      51 - noseSneerRight
      52 - tongueOut

      Hand gestures
      ["None", "Closed_Fist", "Open_Palm", "Pointing_Up", "Thumb_Down", "Thumb_Up", "Victory", "ILoveYou"]

      • Misaki hat auf diesen Beitrag geantwortet.

        • Mario

        • Beitrag Nr. 122
        • Dieser Beitrag ist in Englisch

        @SilverStraw oh, a new MediaPipe model? That's super cool! Likely worth the time investment.

        10 Tage später

        Misaki https://silverstraw.itch.io/spine-vtuber-prototype-2
        I need your help breaking it the application again 🤣. This is using the new facing tracking. I haven't added any tracking smoothing and I am not sure if it is needed. You can tell me your thoughts.

        Hopefully I can transfer the body animation code to the new body tracking without much problem like the face animation. The draw order won't be ready with the body tracking because I haven't figure a good solution to handle depth for Spine skeletons.

        • Misaki hat auf diesen Beitrag geantwortet.

            SilverStraw The new face tracking seems very fine! 😃 You mentioned you did not add any tracking smoothing, but it already seems very smooth. The following video is the result of testing on my end:

            It may be hard to tell from the video, but I feel that the movement has become less wobbly, and when I want to stop the movement, it can be stopped properly.

            I'm looking forward to body tracking becoming available! 💓

              11 Tage später

              2.0.1

              • Added a check box for enabling body pose tracking. It is located under "AI Settings" menu > "AI Tracking Modes" section > "Body Tracking".
              • Added a button for 3d plot graph showing body pose. It is named "Update 3D Plot" and is located under "Camera Settings" menu.

              https://silverstraw.itch.io/spine-vtuber-prototype-2

              Test model updated.
              https://silverstraw.itch.io/spine-vtube-test-model

                SilverStraw Awesome!! I tried it with your test model and it certainly did body tracking on my PC. It's absolutely fun to have the model on the screen react to my movements 😄 When I get a chance, I would like to modify my model for body tracking. Great job!! 🎉

                  • Misaki

                  • Beitrag Nr. 127
                  • Bearbeitet
                  • Dieser Beitrag ist in Englisch

                  For people who don't have time to try it for yourself, I've recorded a screen video. This may be helpful for those who are having trouble understanding how to set it up:

                  For your information, I was wearing a skirt when I recorded this, and I think that's why the tracking of my legs wasn't working very well.

                  @Misaki I noticed that this body tracking works better when the whole body is within frame from my experience. It may not completely be your skirt's fault. If you have the chance could you test the body tracking wearing a skirt but have your foot within web camera frame. I think the body tracking would predict better results that way.

                  The body tracking software makes more assumptive predictions, like when your legs and arms are not in frame.

                  I might have the software check both if the knee and ankle are out of frame and prevent them from going swaying wildly. So far, the software checks those features individually.

                  I mentioned this on Discord. The next tracking I should tackle is hand gesture since I think it's straightforward. Various hand gestures: closed fist, open palm, pointing up, thumbs down, thumbs up, victory sign, I❤️U sign. If the tracker detects any of the hand gestures with a high degree than swap out the hand components. I hope it is able to distinguish left and right hands.

                  • Mario

                  • Beitrag Nr. 129
                  • Dieser Beitrag ist in Englisch

                  Just chiming in to say this is super cool! I should play with the new MediaPipe models one day as well and resurrect my vtubeing app 🙂

                  10 Tage später

                  2.0.2

                  • Add a pointing downward behavior for the Spine model legs when the tracked legs are outside the video frame boundary. The leg positions are more assumptive and less reliable when outside of video frame boundary. The legs should default to pointing downward ( standing position ) to look more natural than flailing legs.
                  • Start to implement web parallel processing for browsers that support it. Multiple threads might speed up computation tasks.
                  • Defer 3d graph plotting task to optimize canvas rendering.
                  • Save 3d plot graph camera location when user rotates the graph. Note that the user can only rotate the graph when the graph is not being updated ("Update 3D Plot" is unchecked).
                  • Replace the word "body" with "torso" in the animation names of "body_rotate", "body_scale_y", and "body_scale_x". Make these animation names less ambiguous.

                  https://silverstraw.itch.io/spine-vtuber-prototype-2

                  Test model updated.
                  https://silverstraw.itch.io/spine-vtube-test-model

                  16 Tage später

                  Spine Vtuber Prototype 2.0.3

                  • Added a check box for enabling hand tracking. It is located under "AI Settings" menu > "AI Tracking Modes" section > "Hand Tracking".
                  • Added "Debug Bounding Boxes" checkbox under "Canvas Settings" menu. Axis aligned bounding box and bounding polygon have separate color and opacity settings. Bounding polygons would help visualize collision shapes.
                  • Fixed a lag bug caused from turning on and off the camera/video. The software activated body tracking too frequently in a short amount of time. Mitigated by slowing down the activation from microseconds to milliseconds.

                  https://silverstraw.itch.io/spine-vtuber-prototype-2

                  Spine Vtuber Prototype 2.0.4
                  Implemented hand gesture tracking to the application. Two hands closer to the goal.

                  https://www.deviantart.com/silverstraw/art/Spine2D-Vtuber-Hand-Gesture-Test-974400292

                  • Fixed "Showing Landmarks" canvas bug where it was not showing. The canvas was erasing faster than it can draw due to the 2.0.3 lag bug. Synchronized the erasing only when it is ready to draw the next frame.
                  • Added hand gesture tracking for closed fist, point up, victory sign, I love you sign, thumbs up. thumbs down. Hands open is assumed to be the default hand position.

                  https://silverstraw.itch.io/spine-vtuber-prototype-2
                  https://silverstraw.itch.io/spine-vtube-test-model

                  • Misaki hat auf diesen Beitrag geantwortet.

                    SilverStraw This is definitely AMAZING!! 😍 As soon as I have time, I would like to modify my model to support hand tracking and try it out.

                      I updated and grouped the Spine animation names. The grouping is necessary as this project has grown to include various optional tracking modes.

                      https://itch.io/t/2951812/spine-animation-names#post-8031587

                      Standby Animations

                      1. idle
                      2. breathe

                      Face Animations

                      1. face_roll_left
                      2. face_roll_right
                      3. face_pitch_down
                      4. face_pitch_up
                      5. face_yaw_left
                      6. face_yaw_right
                      7. left_brow_raise
                      8. left_eye_open
                      9. left_pupil_pitch_down
                      10. left_pupil_pitch_up
                      11. left_pupil_yaw_left
                      12. left_pupil_yaw_right
                      13. mouth_height
                      14. mouth_width
                      15. right_brow_raise
                      16. right_eye_open
                      17. right_pupil_pitch_down
                      18. right_pupil_pitch_up
                      19. right_pupil_yaw_left
                      20. right_pupil_yaw_right

                      Body Animations

                      1. torso_scale_y
                      2. torso_scale_x
                      3. torso_rotate
                      4. left_upper_arm_scale
                      5. left_upper_arm_rotate
                      6. left_lower_arm_scale
                      7. left_lower_arm_rotate
                      8. right_upper_arm_scale
                      9. right_upper_arm_rotate
                      10. right_lower_arm_scale
                      11. right_lower_arm_rotate
                      12. left_upper_leg_scale
                      13. left_upper_leg_rotate
                      14. left_lower_leg_scale
                      15. left_lower_leg_rotate
                      16. right_upper_leg_scale
                      17. right_upper_leg_rotate
                      18. right_lower_leg_scale
                      19. right_lower_leg_rotate

                      Hand Gesture Animations

                      1. left_close_fist
                      2. right_close_fist
                      3. left_point_up
                      4. right_point_up
                      5. left_victory
                      6. right_victory
                      7. left_iloveyou
                      8. right_iloveyou
                      9. left_thumb_up
                      10. right_thumb_up
                      11. left_thumb_down
                      12. right_thumb_down