From 4fa2471b9cb78aebc58a1de0ab1864f7efb0c8b1 Mon Sep 17 00:00:00 2001 From: Emiliana <38952746+emilianavt@users.noreply.github.com> Date: Sat, 8 Feb 2020 21:34:36 +0100 Subject: [PATCH] Information about the expression detection component --- README.md | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 93fe6f2..6b706ae 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,27 @@ The `OpenSeeLauncher` component uses WinAPI job objects to ensure that the track Additional custom commandline arguments should be added one by one into elements of `commandlineArguments` array. For example `-v 1` should be added as two elements, one element containing `-v` and one containing `1`, not a single one containing both parts. +## Expression detection + +The `OpenSeeExpression` component can be added to the same component as the `OpenSeeFace` component to detect specific facial expressions. It has to be calibrated on a per-user basis. It can be controlled either through the checkboxes in the Unity Editor or through the equivalent public methods that can be found in its source code. + +To calibrate this system, you have to gather example data for each expression. The way `OpenSeeExpression` is set up, it requires 400 examples of each expression, which should take around 30 seconds to capture. If capture is too fast, you can use the `recordingSkip` option to slow it down. + +The general process is as follows: + +* Type in a name for the expression you want to calibrate. +* Make the expression and hold it, then tick the recording box. +* Keep holding the expression and move your head around and turn it in various directions. +* After a short while, start talking while doing so if the expression should be compatible with talking. +* When the "Percent Recorded" field is at around 50, untick the recording box and work on capturing another expression. +* When you have them all at around 50, go back to the first and go through them again to bring all the way to 100%. +* Then tick the train box and you should get some statistics in the lower part. +* Tick the predict box and it will show you which expression you are making. + +To delete the captured data for an expression, type in its name and tick the "Clear" box. To save both the trained model and the captured training data, type in a filename including its full path in the "Filename" field and tick the "Save" box. To load it, enter the filename and tick the "Load" box. + +Up to 25 expressions are supported, but a more reasonable number is 5-6. + # General notes * The tracking seems to be quite robust even with partial occlusion of the face, glasses or bad lighting conditions. @@ -48,11 +69,14 @@ Additional custom commandline arguments should be added one by one into elements # Models -Three pretrained models are included. Using the `--model` switch, it is possible to select them for tracking. +Four pretrained models are included. Using the `--model` switch, it is possible to select them for tracking. -* Model **0**: This is the original model I trained. It is the most reliable for general 3D pose tracking, but it has difficulties with eyebrow and eye positions. -* Model **1** (default): This model trades off a little bit of robustness for better facial feature tracking. -* Model **2**: This is a smaller model that can run much faster, easily reaching 60 - 90 fps on a single CPU core. However, the tracking robustness and accuracy are lower. +* Model **0**: This is a very fast, low accuracy model based on ShuffleNetV2. (67fps) +* Model **1**: This is a slightly slower model with better accuracy based on MobileNetV3. (62fps) +* Model **2**: This is a slower model with good accuracy based on ShuffleNetV2. (45fps) +* Model **3** (default): This is the slowest and highest accuracy model. It is based on MobileNetV3. (42fps) + +FPS measurements are from running on one core of my CPU. # Release builds @@ -97,8 +121,6 @@ Additional training has been done on the WFLW dataset after reducing it to 66 po year = {2018} } -The original model can still be selected with `--model 0` instead. - For the training the gaze and blink detection model, the WFLW dataset and [MRL Eye](http://mrl.cs.vsb.cz/eyedataset) datasets were used. ## Algorithm @@ -110,16 +132,19 @@ The algorithm is inspired by: * [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) by Olaf Ronneberger, Philipp Fischer, Thomas Brox * [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://arxiv.org/abs/1704.04861) by Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam * [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://arxiv.org/abs/1807.11164) by Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, Jian Sun +* [Searching for MobileNetV3](https://arxiv.org/abs/1905.02244) by Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, Hartwig Adam -The ShuffleNet V2 code is taken from `torchvision`. +The ShuffleNet V2 code is taken from `torchvision`. The MobileNetV3 code was taken from [here](https://github.com/rwightman/gen-efficientnet-pytorch). For all training after the first model, a modified version of [Adaptive Wing Loss](https://github.com/tankrant/Adaptive-Wing-Loss) was used. * [Adaptive Wing Loss for Robust Face Alignment via Heatmap Regression](https://arxiv.org/abs/1904.07399) by Xinyao Wang, Liefeng Bo, Li Fuxin +For expression detection, [LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) is used. + # License -The code and model are distributed under the BSD 2-clause license. +The code and models are distributed under the BSD 2-clause license. `models/haarcascade_frontalface_alt2.xml` is distributed under its own license. You can find licenses of third party libraries used for binary builds in the `Licenses` folder.