How To Draw Lego Lip Sync Faces

---

---

Rhubarb Lip Sync allows you to quickly create second mouth blitheness from voice recordings. Information technology analyzes your audio files, recognizes what is being said, and then automatically generates lip sync data. You tin can use it for animative speech in estimator games, animated cartoons, or any similar projection.

Rhubarb Lip Sync integrates with the following applications:

• Adobe After Effects (run into below)

• Moho and OpenToonz (encounter below)

• Spine by Esoteric Software (see beneath)

• Vegas Pro by Magix (run into beneath)

• Visionaire Studio (encounter external link)

In improver, you can use Rhubarb Lip Sync'southward command line interface (CLI) to generate files in various output formats (TSV/XML/JSON).

Demo video

Click the epitome for a demo video.

Integrations

Adobe Subsequently Effects

Y'all tin apply Rhubarb Lip Sync to breathing dialog right from Adobe Later Effects. For more than data, follow this link or see the directory extras/AdobeAfterEffects.

Moho and OpenToonz

Rhubarb Lip Sync tin create .dat switch data files, which are understood by Moho and OpenToonz. You lot can prepare the frame rate using the --datFrameRate selection; to control the shape names, use the --datUsePrestonBlair flag. For more details, run across Command-line options.

Spine by Esoteric Software

Rhubarb Lip Sync for Spine is a graphical tool that allows you to import a Spine project, perform automatic lip sync, then re-import the event into Spine. For more data, follow this link or see the directory extras/EsotericSoftwareSpine of the download.

Vegas Pro by Magix

Rhubarb Lip Sync also comes with two plugin scripts for Vegas Pro (previously Sony Vegas). For more information, follow this link or meet the directory extras/MagixVegas of the download.

Oral fissure shapes

Rhubarb Lip Sync can use between half dozen and nine different mouth positions. The first half-dozen oral fissure shapes (Ⓐ-Ⓕ) are the basic mouth shapes and the absolute minimum you take to depict for your character. These six oral fissure shapes were invented at the Hanna-Barbera studios for shows such every bit Scooby-Doo and The Flintstones. Since so, they have evolved into a de-facto standard for 2d animation, and have been widely used by studios like Disney and Warner Bros.

In addition to the six bones mouth shapes, at that place are three extended mouth shapes: Ⓖ, Ⓗ, and Ⓧ. These are optional. You may choose to draw all iii of them, selection simply one or ii, or leave them out entirely.

Ⓐ		Closed oral fissure for the "P", "B", and "M" sounds. This is nigh identical to the Ⓧ shape, but there is ever-so-slight pressure between the lips.
Ⓑ		Slightly open mouth with clenched teeth. This mouth shape is used for nearly consonants ("K", "S", "T", etc.). It's also used for some vowels such as the "EE" audio in bee.
Ⓒ		Open up rima oris. This mouth shape is used for vowels like "EH" as in men and "AE" as in bat. It'due south also used for some consonants, depending on context. This shape is also used as an in-between when animating from Ⓐ or Ⓑ to Ⓓ. Then make sure the animations ⒶⒸⒹ and ⒷⒸⒹ look smooth!
Ⓓ		Wide open up mouth. This mouth shapes is used for vowels like "AA" as in father.
Ⓔ		Slightly rounded rima oris. This oral fissure shape is used for vowels similar "AO" equally in off and "ER" equally in bird. This shape is also used as an in-between when animating from Ⓒ or Ⓓ to Ⓕ. Make certain the rima oris isn't wider open than for Ⓒ. Both ⒸⒺⒻ and ⒹⒺⒻ should event in smoothen blitheness.
Ⓕ		Puckered lips. This mouth shape is used for "UW" as in you, "OW" as in show, and "Due west" equally in way.
Ⓖ		Upper teeth touching the lower lip for "F" as in for and "V" as in 5ery. This extended oral fissure shape is optional. If your art mode is detailed enough, it greatly improves the overall look of the blitheness. If y'all determine not to utilise it, yous can specify and so using the extendedShapes selection.
Ⓗ		This shape is used for long "50" sounds, with the natural language raised behind the upper teeth. The oral fissure should be at least far open up every bit in Ⓒ, only not quite equally far as in Ⓓ. This extended mouth shape is optional. Depending on your fine art fashion and the angle of the head, the natural language may not exist visible at all. In this case, there is no bespeak in drawing this extra shape. If yous make up one's mind non to use it, you can specify so using the extendedShapes pick.
Ⓧ		Idle position. This mouth shape is used for pauses in speech. This should exist the same mouth drawing you use when your character is walking effectually without talking. It is almost identical to Ⓐ, only with slightly less pressure between the lips: For Ⓧ, the lips should exist closed only relaxed. This extended oral cavity shape is optional. Whether there should be any visible difference between the rest position Ⓧ and the closed talking oral fissure Ⓐ depends on your art style and personal taste. If you decide not to use it, y'all can specify so using the extendedShapes option.

How to run Rhubarb Lip Sync

Full general usage

Rhubarb Lip Sync is a command-line tool that is currently available for Windows and Os 10.

• Download the latest release and unzip the file anywhere on your computer.

• Call rhubarb, passing it an audio file every bit statement and telling information technology where to create the output file. In its simplest form, this might look similar this: rhubarb -o output.txt my-recording.wav. In that location are additional control-line options you can specify in order to become better results.

• Rhubarb Lip Sync will analyze the sound file, animate information technology, and create an output file containing the animation. If an error occurs, Rhubarb Lip Sync will instead print an error message to stderr and exit with a non-zero leave code.

Command-line options

Basic command-line options

The following command-line options are the about common:

Option	Description
<input file>	The audio file to be analyzed. This must exist the last control-line statement. Supported file formats are WAVE (.wav) and Ogg Vorbis (.ogg).
-r <recognizer>, --recognizer <recognizer>	Specifies how Rhubarb Lip Sync recognizes spoken communication within the recording. Options: pocketSphinx (employ for English recordings), phonetic (utilise for non-English language recordings). For details, see Recognizers. Default value: pocketSphinx
-f <format>, --exportFormat <format>	The export format. Options: tsv (tab-separated values, see details), xml (see details), json (run across details), dat (see Moho and OpenToonz). Default value: tsv
-d <path>, --dialogFile <path>	With this option, you can provide Rhubarb Lip Sync with the dialog text to become more reliable results. Specify the path to a obviously-text file (in ASCII or UTF-8 format) containing the dialog independent in the sound file. Rhubarb Lip Sync volition yet perform word recognition internally, merely it volition prefer words and phrases that occur in the dialog file. This leads to better recognition results and thus more than reliable blitheness. For instance, permit's say you're recording dialog for a computer game. The script says: "That's all gobbledygook to me." But actually, the vocalism artist ends upward maxim "That's just gobbledygook to me," deviating from the dialog. If yous specify a dialog file with the original line ("That'due south all gobbledygook to me"), this will still let Rhubarb Lip Sync to produce improve results, because information technology will picket out for the uncommon word "gobbledygook". Rhubarb Lip Sync will ignore the dialog file where it audibly differs from the recording, and benefit from it where it matches. It is always a proficient idea to specify the dialog text. This will usually lead to more than reliable mouth animation, even if the text is not completely accurate.
--extendedShapes <cord>	As described in Mouth shapes, Rhubarb Lip Sync uses six basic mouth shapes and up to three extended oral cavity shapes, which are optional. Use this option to specify which extended oral fissure shapes should be used. For case, to use only the Ⓖ and Ⓧ extended rima oris shapes, specify GX; to employ just the six basic mouth shapes, specify an empty string: "". Default value: GHX
-o, --output <output file>	The name of the output file to create. If the file already exists, information technology will be overwritten. If you don't specify an output file, the event will be written to stdout.
--version	Displays version data and exits.
-h, --help	Displays usage information and exits.
--datFrameRate number	Only valid when using the dat export format. Controls the frame rate for the output file. Default value: 24
--datUsePrestonBlair	Simply valid when using the dat export format. Uses Preston Blair mouth shapes names instead of the default alphabetical ones. This applies the following mapping: Alphabetic proper noun Preston Blair name A MBP B etc C East D AI E O F U Yard FV H L X rest Caution: This mapping is merely applied when exporting, after the recording has been animated. To command which mouth shapes to use, use the extendedShapes option with the alphabetic names. Tip: For optimal results, make certain your mouth drawings follow the guidelines in the Mouth shapes section. This is easier if you stick to the alphabetic names instead of the Preston Blair names. The only situation where y'all need to use the Preston Blair names is when you lot're using OpenToonz, considering OpenToonz but supports the Preston Blair names.

Advanced command-line options

The following command-line options tin exist helpful in special situations, especially when automating Rhubarb Lip Sync.

Pick	Description
-q, --quiet	Past default, Rhubarb Lip Sync writes a number of progress messages to stderr. If you're using information technology as part of a batch process, this may ataxia your console. If you specify the --tranquility flag, there won't be any output to stderr unless an error occurred. You can combine this pick with the consoleLevel pick to change the minimum event level that is printed to stderr.
--machineReadable	This option is useful if y'all desire to integrate Rhubarb Lip Sync with another (possibly graphical) application. All status messages to stderr will be in structured JSON format, allowing your plan to parse them and display a graphical progress bar or something similar. For details, see Machine-readable status messages.
--consoleLevel <level>	Sets the log level for reporting to the panel (stderr). Options: trace, debug, info, alert, error, fatal. If --quiet is also specified, only events with the specified level or higher volition be printed. Otherwise, a small number of essential events (startup, progress, etc.) will be printed fifty-fifty if their levels are below the specified value. Default value: error
--logFile <path>	Creates a log file with diagnostic information at the specified path.
--logLevel <level>	Sets the log level for the log file. But events with the specified level or higher will exist logged. Options: trace, debug, info, warning, error, fatal. Default value: debug
--threads <number>	Rhubarb Lip Sync uses multithreading to speed up processing. Past default, it creates equally many worker threads as there are cores on your CPU, which results in optimal processing speed. You may choose to specify a lower number if y'all experience that Rhubarb Lip Sync is slowing down other applications. Specifying a higher number is not recommended, equally it won't result in whatsoever additional speed-upwardly. Notation that for short audio files, Rhubarb Lip Sync may choose to utilise fewer threads than specified. Default value: as many threads as your CPU has cores

Recognizers

The offset step in processing an sound file is determining what is beingness said. More specifically, Rhubarb Lip Sync uses speech recognition to effigy out what audio is beingness said at what bespeak in time. You can choose between 2 recognizers:

PocketSphinx

PocketSphinx is an open-source speech recognition library that generally gives good results. This is the default recognizer. The downside is that PocketSphinx only recognizes English dialog. And then if your recordings are in a language other than English, this is not a proficient choice.

Phonetic

Rhubarb Lip Sync also comes with a phonetic recognizer. Phonetic means that this recognizer won't try to sympathize entire (English) words and phrases. Instead, it will recognize individual sounds and syllables. The results are ordinarily less precise than those from the PocketSphinx recognizer. The reward is that this recognizer is language-independent. Utilize information technology if your recordings are not in English.

Output formats

The output of Rhubarb Lip Sync is a file that tells y'all which mouth shape to display at what time within the recording. You tin can choose between three file formats — TSV, XML, and JSON. The following paragraphs show you what each of these formats looks like.

Tab-separated values (tsv)

TSV is the simplest and nearly compact export format supported by Rhubarb Lip Sync. Each line starts with a timestamp (in seconds), followed by a tab, followed by the name of the oral fissure shape. The following is the output for a recording of a person saying 'Hullo.'

                    0.00	Ten 0.05	D 0.27	C 0.31	B 0.43	X 0.47	X                  

Here's how to read information technology:

• At the beginning of the recording (0.00s), the oral fissure is closed (shape Ⓧ). The very first output will always have the timestamp 0.00s.

• 0.05s into the recording, the mouth opens broad (shape Ⓓ) for the "HH" audio, anticipating the "AY" sound that will follow.

• The second one-half of the "AY" diphtong (0.31s into the recording) requires clenched teeth (shape Ⓑ). Before that, shape Ⓒ is inserted as an in-between at 0.27s. This allows for a smoother animation from Ⓓ to Ⓑ.

• 0.43s into the recording, the dialog is finished and the mouth closes again (shape Ⓧ).

• The last output line in TSV format is special: Its timestamp is ever the very finish of the recording (truncated to a multiple of 0.01s) and its value is always a closed mouth (shape Ⓧ or Ⓐ, depending on your extendedShapes settings).

XML format (xml)

XML format is rather verbose. The following is the output for a person proverb 'Hi,' the same recording as above.

<?xml                                          version=                      "i.0"                                                              encoding=                      "utf-8"                    ?> <rhubarbResult>   <metadata>     <soundFile>C:\Users\Daniel\Desktop\av\hi\hi.wav</soundFile>     <duration>0.47</elapsing>   </metadata>   <mouthCues>     <mouthCue                    start=                      "0.00"                                        end=                      "0.05"                    >X</mouthCue>     <mouthCue                    start=                      "0.05"                                        end=                      "0.27"                    >D</mouthCue>     <mouthCue                    start=                      "0.27"                                        terminate=                      "0.31"                    >C</mouthCue>     <mouthCue                    start=                      "0.31"                                        cease=                      "0.43"                    >B</mouthCue>     <mouthCue                    first=                      "0.43"                                        end=                      "0.47"                    >10</mouthCue>   </mouthCues> </rhubarbResult>

The file starts with a metadata block containing the total path of the original recording and its elapsing (truncated to a multiple of 0.01s). Subsequently that, each mouthCue element indicates the start and stop of a certain oral fissure shape, as explained for TSV format. Note that the end of each mouth cue is identical with the start of the following one. This is a bit redundant, but it ways that nosotros don't demand a special terminal element similar in TSV format.

JSON format (json)

JSON format is very similar to XML format. The choice mainly depends on the programming language you utilize, which may have built-in support for one format but non the other. The following is the output for a person saying 'Hullo,' the same recording equally above.

{                    "metadata": {                    "soundFile":                                          "C:\\Users\\Daniel\\Desktop\\av\\hello\\hi.wav"                    ,                    "elapsing":                    0.47                    },                    "mouthCues": [     {                    "start":                    0.00,                    "end":                    0.05,                    "value":                                          "X"                    },     {                    "start":                    0.05,                    "end":                    0.27,                    "value":                                          "D"                    },     {                    "start":                    0.27,                    "end":                    0.31,                    "value":                                          "C"                    },     {                    "start":                    0.31,                    "stop":                    0.43,                    "value":                                          "B"                    },     {                    "start":                    0.43,                    "end":                    0.47,                    "value":                                          "X"                    }   ] }

There is nothing surprising hither; everything said about XML format applies to JSON, besides.

Machine-readable condition letters

Use the --machineReadable control-line option to enable motorcar-readable condition letters. In this mode, each line printed to stderr volition exist an object in JSON format. Every object contains the following:

• Property type: The type of the event. Currently, ane of "start" (application starting time), "progress" (numeric progress), "success" (successful termination), "failure" (unsuccessful termination), and "log" (a log message without structured information).

• Consequence-specific structured data. For case, a "progress" event contains the property value with a numeric value between 0.0 and 1.0.

• Holding log: A log message describing the event, plus severity information. If you aren't interested in the structured information, you can brandish this as a fallback. For example, a "progress" event with the structured information "value": 0.69 may contain the following redundant log message: "Progress: 69%".

You can combine this option with the consoleLevel option. Note, however, that this just affects unstructured events of type "log" (not to be confused with the log property each issue contains).

The following is an example output to stderr from a successful run:

{                  "type":                                      "showtime"                  ,                  "file":                                      "hello.wav"                  ,                  "log": {                  "level":                                      "Info"                  ,                  "message":                                      "Application startup. Input file:                    \"hi.wav\"."                  } } {                  "type":                                      "progress"                  ,                  "value":                  0.00,                  "log": {                  "level":                                      "Trace"                  ,                  "bulletin":                                      "Progress: 0%"                  } } {                  "type":                                      "progress"                  ,                  "value":                  0.01,                  "log": {                  "level":                                      "Trace"                  ,                  "message":                                      "Progress: ane%"                  } } {                  "type":                                      "progress"                  ,                  "value":                  0.03,                  "log": {                  "level":                                      "Trace"                  ,                  "message":                                      "Progress: 3%"                  } } {                  "blazon":                                      "progress"                  ,                  "value":                  0.06,                  "log": {                  "level":                                      "Trace"                  ,                  "message":                                      "Progress: 6%"                  } } {                  "blazon":                                      "progress"                  ,                  "value":                  0.69,                  "log": {                  "level":                                      "Trace"                  ,                  "message":                                      "Progress: 68%"                  } } {                  "type":                                      "progress"                  ,                  "value":                  1.00,                  "log": {                  "level":                                      "Trace"                  ,                  "message":                                      "Progress: 100%"                  } }                  // Result information, printed to stdout...                  {                  "type":                                      "success"                  ,                  "log": {                  "level":                                      "Info"                  ,                  "message":                                      "Application terminating normally."                  } }

The following is an example output to stderr from a failed run:

{                  "type":                                      "start"                  ,                  "file":                                      "no-such-file.wav"                  ,                  "log": {                  "level":                                      "Info"                  ,                  "message":                                      "Awarding startup. Input file:                    \"no-such-file.wav\"."                  } } {                  "type":                                      "failure"                  ,                  "reason":                                      "Fault processing file                    \"no-such-file.wav\".\nCould non open up sound file                    \"no-such-file.wav\".\nNo such file or directory"                  ,                  "log": {                  "level":                                      "Fatal"                  ,                  "message":                                      "Application terminating with mistake: Error processing file                    \"no-such-file.wav\".\northCould not open audio file                    \"no-such-file.wav\".\nNo such file or directory"                  } }

Note that the output format adheres to SemVer. That means that the JSON output created after a minor upgrade will still exist compatible. Note, however, that the post-obit kinds of changes may occur at whatever time, considering I consider them not-breaking:

• Additional types of progress events. Just ignore those events whose types you do not know or apply their unstructured log belongings.

• Additional properties in any object. Just ignore properties yous aren't interested in.

• Changes in JSON formatting, such as a re-ordering of properties or changes in whitespaces (except for line breaks — every event will remain on a singe line)

• Fewer or more than events of blazon "log" or changes in the wording of log messages

Versioning (SemVer)

Rhubarb Lip Sync uses Semantic Versioning (SemVer) for its command-line interface. For full general information on Semantic Versioning, take a look at the official SemVer website.

As a rule of thumb, everything you can utilise through the command-line interface adheres to SemVer. Everything else (i.e., the source code, integrations with third-party software, etc.) does not.

I'd love to hear from you!

Take you lot created something great using Rhubarb Lip Sync? —Permit me know on Twitter or send me an email at dwolf@dannad.de!

Do yous need assist? Accept yous spotted a bug? Do you lot accept a proffer? —Create an issue!

---

JetBrains take been kind plenty to supply me with a free Open Source license of ReSharper C++.

Source: https://github.com/DanielSWolf/rhubarb-lip-sync

Posted by: solisviturts.blogspot.com

Share this post