Lipscripts for 3d max

For 3ds Max Users, we have developed a maxscript that reads in Lipsync Tool documents and creates animations within max. This software is included in the Lipsync Tool.

Download it: Lipscript_V1.ms

The Lipscript is a maxscript used to integrate the data generated by the Lipsync Tool into 3ds max. The lipscript reads .anno files and creates animations in max based on the lipsync data and the user configuration.

The goal is flexibility. The Lipsync tool outputs excellent lipsync data, but getting that data into YOUR rig is the big thing. We didn't want to do a plug-in because that prevents technical artists from doing their thing. A maxscript, in source form, is the best way to make the thing customizable.

Out of the box, the Lipscript can be used to create Lipsync for :

• Morph based Facial Setups

• Bone based Facial Setups

• Setups using a combination of both, and also

• Facial rigs run by spinner based custom Attributes, sliders, etc.

Lipsync Tool is capable of recognizing 40 different phonemes. Using Lipscript, you can combine these phonemes in any order to make different visemes, which then can be mapped to a facial setup, be it morph-based, bone-based or custom-attribute-based.

For example, you could either map "m", "b" and "p" to separate morph targets/sliders/bone Controls or you could club all 3 together and map the combined viseme. Same for all the 40 phonemes. Send us an e-mail to request an "export/save" enabled trial version for free to try this out.

LipScript is used to integrate the data generated by Lipsync Tool (during export) into 3DS Max, so as to be able to generate lipsync for any 3D character model.

The Lipscript Interface is divided into 2 rollouts:
1. User Defined Phoneme Combinations Rollout and
2. Apply Lipsync Rollout.


LIPSCRIPT UI

Basic Usage Info:

1. The first step is to export lipsync from within Lipsync Tools. This generates a .anno file which contains complete information regarding the phonemes recognized by the software, along with their intensities and timing.

2. Now, open the max file where you have the character onto which the lipsync is to be applied, and run Lipscript.

3. In the phoneme Combinations section of the UI, the user is provided with editable text boxes, as well as drop down lists where he can combine phonemes as per his preference. You can either type phonemes (separated by a comma) into the text-edit boxes or you can choose multiple phonemes from the drop downs provided along side.

Currently, the script provides 20 such rows, meaning you can make 20 different phoneme combinations. The phonemes in each row will make up a viseme. for example, typically, the phonemes "m", "b", and "p" should be clubbed together to form a single viseme, since the shape of the mouth assumes a similar pose while uttering any of these 3 sounds.

These user combinations can also be saved as presets for re-use later on.

4. Once the combinations have been defined, its time to map these visemes to your character's facial setup.

5. Lipscript can be used to create Lipsync for :
#Morph based Facial Setups,
#Bone based Facial Setups,
#Setups using a combination of both, and also
#Facial rigs run by spinner based custom Attributes, sliders, etc.

6. For a character with morph target based facial setup, click on the button "Select Object with Morpher" and then select the character head (You will be able to select only objects with a morpher modifier in its modifier stack.) This action will activate dropdown lists in the 'Viseme rows', in the Morph Target Mapping section. Each of these drop down lists will contain the morph targets that have been assigned to the character.
Make appropriate morph-target selections in the drop downs corresponding to the phoneme combination that you have made in each row.


7. For a Bone based facial setup, the user has to select all the control objects which will be used to animate the face. Please note that the script provides you with a feature which automizes the task of generating multiple position lists on each of these control objects corresponding to each of the user-defined visemes. After selecting all the required control objects, click on "Select All Control Objects" button. This activates the button " Setup Position Lists" ..... a click here changes the current position controller of each of these objects to
position list, and then it adds multiple "Position XYZ" controllers in the available positions.

These position controllers will be named 'AnnoViseme1', 'AnnoViseme2' ..... and so on corresponding to the used viseme slots in the script UI. The number of controllers in the position list at the end of this action should be one more than the number of visemes that you have setup.

The one additional controller, placed at the top of the list, can be used to manually keyframe the control objects where needed.

Important:
While using Lipscript for a bone based Facial Rig, it is necessary to first create the phoneme combinations. This is because, Lipscript creates position lists based on the number of used Viseme Slots.

The next step in a bone based facial setup is to make appropriate mouth-shapes - corresponding to each viseme. This can be easily done by alternatively clicking on each of the checkboxes named "Shape Viseme #" and then moving the control objects as required, for the corresponding viseme. These checkbuttons can also be used after generating lipsync. This means that you can modify the mouth shapes
before and after lipsync.


8. Load Map/ Save Map:
A Load/Save Mapping feature is provided both for morph as well as Bone based facial setups.

Tip:
All dialogue audio may not necessarily have all the 40 phonemes which Lipsync Tool is able to recognize.
Some dialogues, for example, may contain only 10 or 15. So, while mapping the visemes to either bone or morph rigs, you only need to map the phonemes which are present in the current audio file. Lipscript provides you with a feature by which you can visualize which phonemes you actually need to map, for the current audio file. Click on "Choose exported anno file for reference" and select the .anno file which you have exported from Lipsync Tool. All the phonemes found in this .anno file will be displayed in the editbox on top of the rollout, along with the number of phonemes found.

step 6 or 7 completes the Phoneme Mapping section, and we now move onto actually generating keyframes for lipsync, in the 'Apply Lipsync' rollout.




9. Based on your facial setup whether it is morph-target based or pos-list/bone based, choose one of the radio buttons under 'Apply Lipsync To'

10. Choose the .anno file which you had previously exported from Lipsync Tool. This activates the 'SYNC' button.

11. Click 'SYNC'

Note :
By default, the keyframes are generated on 'ticks' (sub-frame level). If you require keyframes to occur on frames (integers) you should select the radio button 'shift keys to occur at frames' in the Import Options.

Also if you need the keyframes to be offset by say, 100 frames, adjust the Start time slider in the UI to 100. This also needs to be done before clicking on 'SYNC'. For example, if you have already generated lipsync of the first dialogue for a character, and now you need to add more lipsync on him with another dialogue.

Choosing the "Custom Attribute" radio option under "Apply Lipsync To" and then proceeding to step 10 and 11 will create a new helper object. This object will have Custom Attribute spinners named "Viseme_1", Viseme_2" and so on, corresponding to the Visemes that you have defined in Step 3.
These spinners will be animated as per the imported data. These animated spinners can be then used (thro wire or reaction controls) to apply lipsync data to any parameter in you facial setup.