4.4 KiB
This is kind of a personal project, but it spiraled out of control and I figured I'd share it in case anyone else can make use of it.
I make no promises to the quality of this code or the reliablity of it
I am not responsible if this somehow causes you issues
I have drastically restructured this, but I'm listing it as a fork because this is still based on emilianavt's work.
I am specifically targeting Vtube Studio on Linux, some stuff was changed to make it work with VTS, idk how well it works with VeeSeeFace at this point
This is also very much tuned to how I use it and what works best for me, so I make no promises there
If you do use it, be aware that I removed directshow support(can't make use of it) and MJPEG is kinda janky, so your best bet is 480p which defaults to raw rgb
This will do a rock steady 30fps on my system as long as my cpu isn't completely overloaded
I'll update this readme to explain more later, but first I need to actually upload all my changes. I didn't expect this to turn into something I wanted to upload.
Packages you'll need
Numpy
OpenCV-python
onnxruntime
Changes (in no particular order):
-Major restructure
-I've broken out major chunks of functionality into separate files to make them easier to deal with and read
---I've tried to break functions down into multiple smaller parts when possible
---I've taken a more object oriented approach in some places
---I've removed features that didn't fit my needs
---Removed functionlaity for tracking multiple faces
---Removed options I didn't forsee myself using (this was intended to be just for me)
-Added more threading and multi-processing
---The webcam (and some new image processing) are now on a separate process
---Previews are now a separate process because I didn't want to deal with the related performance weirdness
---Console messages are now handled via a helper thread, idk if it does any good, but it can't hurt
---VTS communication is also handled via a helper thread
-Added image pre-processing
---I'm now applying a gamma curve to the webcam output to make faces more visible
---The gamma curve is calculated after face tracking is done, it uses a copy of the webcam frame and face location data so get the average brightness of the face
---This has drasitcally improved low light performance
---It also lets me run the webcam with no gain
---Removed the separate library that handled the webcam, then broke that off into a separate library again
---The stuff around the webcam is still simpler now, most of the existing cases didn't apply to my use case
-Revamped the way features are calculated and normalized
---This was kind of my original intent, my eyes kept registering as closed when they were not
---Removed features that didn't seem to be used by Vtube Studio
---Added functionality to apply response curves to features
---Removed the calibration period in favor of a system where the limits of features slowly decay towards a center point
---Removed the average from the way featured are normalized
-Added a feature to prevent errant eye movements
---Each eye has an average confidence and standard deviation calculated every frame
---eye movements more than two standard deviations below average are severely restricted
---The restriction is based on how far the confidence is from the cutoff point
-Added early exits to situations where subsequent steps will fail
-The main process skips frames when the webcam falls behind, once webcam latency is long enough to be late it will remain behind otherwise
-Added warnings to various states such as late webcam frames, longer than ideal frames, and early exits
-Changed which stats are shown on exit, added some stats that get tracked like webcam latency
-Added conditions to skip steps when they are unnecessary, such as not rotating the eye images for small angles
For documentation on how openseeface works and the computer vision models, go see https://github.com/emilianavt/OpenSeeFace I don't want to leave documentation here that makes it seem like this is all my work
Seriously, emilianavt did all the hard work
I was going to do more documentation but github's markdown does not work in any sensical way
idk, I'll add a readme or something, but I tried to make install instructions and it randomly started bolding list items