Hello, sorry for a smaller build this time, I just want to push out some fixes and tweaks rather than waiting on some bigger things to be finished first, particularly some regressions for the facial tracking blendshape auto-detection.

There's a few other smaller additions too, for example you can adjust the volume of people outside of your whisper bubble now.

Some smaller improvements and progress to the new desktop mode as well, with key bindings and legacy dialogs now working properly. I'll have more soon!

- Added "Whisper Voice Volume (outside bubble)" setting, which allows configuring the voice volume of users outside the whisper bubble (requested by @Robyn (QueenHidi))
- Added "StrengthMultiplier" to the AvatarRawMouthData, allowing scaling the strength of all the properties

- Added TongueRaise and TongueExtend expressions to the AvatarExpressionManager
-- Those are useful when there are two separate blendshapes for raising the tongue and then extending it out
-- The heuristics was updated to map the TongueLongStep1 and TongueLongStep2 to these two, resulting in better behavior for models following the HTC Facial Tracker naming convention

- AvatarExpressionManager heuristics now checks for the exact expression naming of the blendshapes first, before using heuristics (based on feedback by @GearBell)
-- Note that for this to work, you must name the blendshapes exactly the same as the expression names you see in the inspector, without any extra spaces, periods or underscores. It is however case insensitive

- Keyboard bindings that use Ctrl, Shift and Alt keys will now also accept the right Ctrl/Shift/Alt keys (based on requested by @Psychpsyo)
- Legacy dialogs (e.g. Hyperlink warning, Create New Directory, Credits Transfer and so on) will now open in the overlay in the new desktop mode so they can be easily interacted with
-- Note that you cannot grab and move them around, only click to interact. This might not be implemented before they're replaced with more modern variants
- Any focused elements (e.g. text fields) will now block keyboard interactions in userspace
-- This fixes the dash opening/closing when using keyboard to type (based on report by @Shadow Panther [RU/EN, UTC+3])
- Text fields will now unfocus when the Escape key is pressed

- Merged Russian locale additions (including MTC Avatar Lobby) by @Shadow Panther [RU/EN, UTC+3]
- Merged Japanese locale addition by @Aesc
- Merged Czech locale addition by @rampa_3 (UTC +1, DST UTC +2)
- Merged Korean locale addition by @MirPASEC

- Fixed regressions in AvatarExpressionDriver, BipedRig and other naming heuristics, causing some blendshapes/bones and/or their chirality to not be detected properly
-- This fixes the HTC blendshape naming convention not automatically setting up all the names anymore
-- If you setup your avatar recently, it's recommended to rerun the auto-assignment
- Fixed headless not measuring auto restart and auto save intervals properly, using a mix of UTC and local time, resulting in incorrect intervals (reported by @Shadow Panther [RU/EN, UTC+3])
- Fixed regression where it wasn't possible to interact with touchables with the same hand that's holding an object with the laser
-- This fixes not being able to invoke the Overwrite menu for world orbs
-- It also fixes not being able to assign references/values to the inspector fields by clicking (reported by @Turk)
- Fixed being able to use worlds that you don't have write permissions for to overwrite your own worlds with (reported by @sirkitree)
- Fixed Unity Native video playback returning audio data when paused
-- This should fix VolumeMeter reporting volume when the source video is paused (reported by @Toxic_Cookie)
-- It should also fix AudioSourceWaveformMesh showing undesirable visuals (reported by @AshtonSparx)

A whole bunch of new goodies for the new desktop mode! You can now rotate items (GMod style!), scale them, scale yourself, take in-game photos (even with timer), save items to inventory and quick switch common tooltips. I've also changed a bunch of the existing bindings too (e.g. the Dash is now opened/closed simply by pressing Esc) and there's a whole bunch of other tweaks, improvements and bugfixes to make the new desktop mode more usable.

Currently it is still usable only by pressing F8 after launching in VR, but I'll be swapping it out within next few days, after adding a few more crucial things. It's coming along!

Also added a bunch of bugfixes to this one (including broken Reflection Probes on Linux, which turned out to be just a stupid typo in a flag when uploading HDR textures to GPU, sorry!) and some regressions (sorry for the inconvenience!). More to come soon!

- Added "Save To Inventory" context menu option, which will save the held object to the current inventory folder
-- Note that you currently need to open inventory folder where you have write permissions first. This will be simplified with the upcoming inventory UI redesign
-- This also allows easily saving items in the new desktop mode

- Added an internal mechanism to load-equip and stash tooltips from an URL
-- Tooltips and other equippable items are now internally tagged as equippable when saved to inventory, which will allow future UI to allow one-click equip directly from inventory
-- When switching tooltips, if the current one is quick loaded, it will be "stashed" - disabled and hidden, preserving its state on the next equip in the same session
- You can now quick switch tooltips in the new desktop mode via tha alphanumeric keys. Bindings are following (the loadout will be customizable in the future):
-- 1 - Dequip
-- 2 - Developer Tip
-- 3 - LogiX Tip
-- 4 - Material Tip
-- 5 - Shape Tip
-- 6 - Light Tip
-- 7 - Grabbable Setter Tip
-- 8 - Character Collider Setter Tip
-- 9 - Microphone
-- 0 - Glue Tip

- You can now capture in-game photos in the new desktop mode
-- To capture instant photo, press Shift+P or Shift+PrintScreen
-- To start timer photo, press Ctrl+P or Ctrl+PrintScreen
-- The capture won't start until you release the combination, allowing you to hold it to see a preview

- You can now rotate held items in the new desktop mode
-- Hold the E key while holding an item and move the mouse around to rotate object around X and Y axes
-- Hold the Shift+E key to rotate only around the vertical axis
- You can now scale held items in the new desktop mode
-- Hold the Shift key while holding an item and use the mouse scroll wheel to scale it

- Added a mechanism to block inputs within InputGroup by specifying a priority level for each binding, simplifying setup of common control schemes (e.g. Ctrl+<Key>, Shift+<Key> and <Key> as modifiers)
- Reworked FingerPhotoGesture into a more general PhotoCaptureManager

- Dash is opened/closed by the Escape key now in the new desktop mode
- The locomotion switch and scaling options are now shown on the context menu even with equipped tooltip
- Keyboard & Mouse input is now fully supressed while the dash is open in the desktop mode
- Lowered the limit of how close you can bring grabbed items to your viewpoint from 10 cm to 5 cm in the new desktop mode
- Increased vertical mouse look angle limit to 89 degrees from 80 to allow looking near straight up and straight down
- You can now use PageUp and PageDown to quickly cycle between active locomotion modules
-- This will have a visual at some later point, not too usable at this point just yet, but it's been added as part of larger work
- You can now self-scale in new desktop mode by holding Ctrl and using the Mouse Scroll button
-- Scaling still needs to be enabled first in the context menu
- Tweaked finger photo gesture supression level, to be more tolerant of bent fingers
- Developer Tooltip laser now considers Gizmo controls as preferred targets and will pass through other objects and stick to them (based on report by @H3BO3)
- Undo & Redo context menu options are not generated when holding items with the active hand

- Merged Japanese locale additions for the new MTC Avatar Lobby by @Aesc
- Merged Czech locale additions for the new MTC Avatar Lobby by @rampa_3 (UTC +1, DST UTC +2)
- Merged Korean locale tweaks by @MirPASEC

- Context menu positioning in the new desktop mode now properly handles user's scale
- Interaction Laser now consider user's current scale when ignoring near raycast hits
-- This fixes the context menu not being usable when the user scale is too small in the new desktop mode
- Fixed SyncType inspector field not accepting any type strings after recent changes (reported by @3x1t_5tyl3)
- Fixed incorrect flags being set for BC6H textures when uploading them to the GPU through OpenGL, resulting in the texture data to be interpreted incorrectly on Linux and Android
-- This fixes ReflectionProbes being overly bright on Linux (@Shifty | Quality Control Lead, @happysmash27, @Polychrome, Silvea12, Enverex and @Develon)
- Fixed favorite avatar, keyboard, camera and home being unfavorited on manual log out (reported by @Flame Soulis and @H3BO3)
- Fixed gamepad mouse look being framerate dependent (reported by @Lewis Snow | Lead Audio Engineer)
- Fixed regression breaking the context menu flick functionality (reported by @Flame Soulis, @DeliriousJax, @ohzee, @H3BO3 and @Turk)
- Fixed regression when aligning laser-held objects to the vertical axis having a small offset from the aligned position, depending on the hand movement speed (reported by @orange)

An extra hotfix to fix some more issues that were found. Compatible with previous build, but update recommended for headless, since it mainly concerns bugs manifesting on those.

- Merged Korean locale translations by @MirPASEC (including new MTC Avatar Lobby)

- Fixed Neos trying to use VR bindings against VirtualController instead of VR one in some cases, causing the user's session to crash
-- This should fix some session crashes when joining worlds or switching locomotion in desktop, particularly with Oculus runtime (reported @epicEaston197, @Shifty | Quality Control Lead, @/home/JellyOsaurusPC)
- Fixed exceptions due to updating InputGroups for owners that have been removed in the input update cycle
- Fixed grabber transform computations using local user's space, instead of the user actually owning the grabber
-- This also fixes exceptions on the headless for the CommonTool, due to not having a user root
- Fixed VRIKAvatar trying to calculate distance to user's head when the head position is not available (e.g. during respawn or on headless)
-- This fixes IK avatars stopping to update when the user exits them on a headless session

Hello everyone, sorry it took a while to get this build ready, but it presents a major milestone in the develpment of the new desktop mode! You can now fully use Context Menus and Equippable items (Tools, Weapons, Gadgets...), making the mode fully usable for core interactions! With those in place, I'll start progress on making this the default desktop mode now. I'll keep the old one as legacy for a bit while it still gets more polish. Right now you still need to startup in VR and press F8 to activate it, but hopefully it'll prove very useful already!

There are some semi-major things that changed/expanded in this internally, the input handling has been restructures quite a bit, which reduced latency of some input handling by a frame (making things feel more responsive), but watch out for potential side effects. Thanks to the following danger testers for catching some game breaking bugs before this went live: @/home/JellyOsaurusPC, @Kulza, @Raith (CytraX), @Alex the pet peeve avali 🐦, @Epsilion, @Gawdl3y and @Shadow Panther [RU/EN, UTC+3]

This build has a whole bunch of tweaks, security improvements and fixes too, so hopefully that'll solve some of the pains and problems that you've been dealing with, sorry that those fixes got held up a while! I hope to have more ready soon, probably tomorrow. I'll probably have to release the weekly update tomorrow too instead of today, it took a bit longer than expected to get this ready and not sure how much longer I'll be able to stay up and focus.

But in the meanwhile, have fun! There's a lot more to come to desktop, but with this we should have all the necessary basics.

- Context Menus are now usable in the new desktop mode
-- Press the Middle mouse button or the "Y" button on gamepad to open/close them
-- They will also open when clicking on certain objects (e.g. Avatars or Tooltips/Equippable items)
-- When Context menu is open, the cursor will switch to a freeform one, allowing to easily select and click on the items without having to aim head at them

- Tools, Gadges and other equippable items are now usable in the new desktop mode
-- When equipped, the avatar's hand is positioned relative to the head so the item is visible in the first person mode
--- The positioning is based on the grip pose. If it's not setup properly, the item will be misaligned
-- The tool/item/gadget is automatically aimed at the point in world currently under the cursor reticle
--- For this to work correctly, the item needs to have a properly setup TipReference, that corresponds to the exact tip and direction that the tool should aim at (using Z axis as forward)
--- Existing RawDataTooltip items are auto-upgraded using heuristics, using sources of raycast or particles. You might need to adjust the TipReference on some manually if the detection doesn't work properly
-- Current bindings are (these are tentative and will likely change)
--- Primary - Left Mouse Button / Right Trigger (gamepad)
--- Secondary - R key / Mouse Button 4 (is present) / X button (gamepad)

- RawDataTooltip & StandardController now provide proper inputs in the desktop mode, making items that rely on those work
-- The values are fed through a VirtualController type from the current input bindings of corresponding actions

- Pointer interaction system now routes the first and second pointer (mouse cursor and/or touch) through the hand interaction system
-- This results in the intereractions posing the avatar's primary and secondary hand and being able to grab and scale with toushcreen
- Neos now renders its own cursor even in freeform cursor mode to provide visual consistency and contextual behavior and avoid the system cursor to be overlaid on top of the in-game reticle
- InputDeviceStreamDriver now disables streams for input devices that are currently inactive, saving network bandwidth
-- E.g. VR controllers when Screen is active and VirtualControllers when VR is active

- You can now self-assign the hearing impaired, color blind, mute and potato badges
-- To assign or remove, simply send following commands to the Neos bot account in your Contact list:
-- /addHearingImpaired, /removeHearingImpaired, /addColorBlind, /removeColorBlind, /addMute, /removeMute, /addPotato, /removePotato

- Restructured update order of the input handling components, avatar posers, VRIK and CommonTool
-- This reduces latency of input response by a frame (e.g. making the laser/cursor respond in the same frame that the input happened, rather than lagging a frame later)
-- It also allows for automatic interactions based on a laser hit (e.g. based on a screen touch that happens instantly)
- Desktop mode grabbing now uses user's head forward direction as rotation reference instead of the hand
-- This reduces slight rotation of the grabbed object upon grabbing due to the hand being procedurally moved
- Cleaned up and tweaked several user related components for driving the root objects based on inputs, tracking compensation and so on
-- Many of the components now implicitly work based on the user they're initialized on, rather than explicitly assigned
-- Where possible, the components only register events for the user actually using them, rather than for everyone
- Improved parsing of URL's when classifying asset types to search the URL query for a file extension in addition to the main data path
-- This makes certain types of URL to get detected properly and imported directly from the clipboard (e.g. IPFS file URL's, as reported by @Komdog)
-- E.g. https://example.com/somepath?file=MyFile.mp4&someparam=42 will now properly import as a video
- Near head hand gestures/shortcuts are no longer triggered in the desktop mode
- Saving world with Ctrl+S is now supressed when the new desktop mode is active (based on report by @Cyro)
- Crouch is now on the Left Bumper on gamepad

- Added new English MTC Avatar Lobby locale strings by @Ryuvi | Technical Artist and @Aegis_Wolf | Art Director

- Merged Estonian locale additions and fixes by @Meron
- Merged Russian locale addition & tweak by @Shadow Panther [RU/EN, UTC+3]
- Merged Korean locale addition by @MirPASEC
- Merged Japanese locale addition by @Aesc
- Merged Esperanto and Chinese locale additions and tweaks by @Melnus
- Merged Czech locale addition by @rampa_3 (UTC +1, DST UTC +2)
- Merged English store page fixes by @rampa_3 (UTC +1, DST UTC +2) and Enverex (will take a bit before it's pushed to store)
- Merged British English addition by Enverex

- Fixed SimpleAvatarProtection throwing an exception when checked against user who has been destroyed (based on report by @3x1t_5tyl3 and @marsmaantje)
- Fixed CommonAvatarBuilder not cleaning up user's default avatar if it fails to be equipped for any reason (based on report by @3x1t_5tyl3 and @marsmaantje)
- Improved security of touch sources (based on report & sample by @Psychpsyo and @Epsilion)
- Improved sanitization of held items being transferred through the userspace (based on report & sample by @Psychpsyo and @Epsilion)
- Reworked mechanism for cleaning up Userspace-only components, so placing them on protected Slots (e.g. Root slot) doesn't cause exceptions and failure to clean them up (based on report by @Psychpsyo)
- Fixed usernames not being sanitized in the World UI and on the world orbs, causing any RTF tags in the username to be parsed (based on report by @Kodey and @Psychpsyo)

- Fixed Neos locking the cursor when running in the VR mode (reported by @Hayden)
- Fixed headless running some actions that should only be ran by user under who is given behavior parented
- Fixed BeforeInputUpdate event being potentially executed twice on some components
- Fixed SmirkLeft and SmirkRight expressions breaking the AvatarExpressionDriver when activated (reported by @Alex the pet peeve avali 🐦)
- Fixed ContextMenu collider being enabled until it's first used
- Fixed some odd interactions when using the new desktop mode with the old one
- Fixed ScreenController update order being same as TrackedDevicePositioner, causing the simulated hand to freak out in some cases (where it ends up initialized after the TrackedDevicePositioner)
- Fixed ValueField<T> and other generic components that handle value types allowing certain non-value types (e.g. System.Type) and causing the session to crash (reported by @dfgHiatus, @Epsilion, @Psychpsyo and @Zyzyl)
- Fixed component exception handling misbehaving when placed on a destroy-protected slot (based on feedback by @Psychpsyo)
- Fixed Neos crashing when running on devices without a mouse present (e.g. Android) (based on report by @かず (kazu / GitHub: kazu0617))
- Fixed reliable user positioning breaking and throwing exception when the user is deleted in the middle of the process, causing other processes to break (based on report by @3x1t_5tyl3 and @marsmaantje)
- Added generic type validation to more components to prevent crashes when creating them with invalid argument (based on findings by @Zyzyl and @Banane9)
- Fixed some World event behaviors (e.g. focus change, on save, user join or leave) breaking due to not setting up proper update scope (e.g. when fields are driven)
- Added Type validation for setting SyncType and related operations, to avoid setting/processing of invalid types (e.g. type with invalid generic argument) and causing the session to crash (reported by @Psychpsyo)
- Fixed LipMoveLeftLower expression on AvatarExpressionDriver driving the target with movement to right, instead of left (reported by @umbran)
- Fixed nameplates using the IsPresentInHeadset state instead of IsPresent, causing them to become semi-transparent when the user switches to the desktop mode after existing their headset (reported by @Shifty | Quality Control Lead)
- Fixed Userspace having two instances of PointerInteractionController, resulting in cursor and touch interactions in userspace being processed twice
Known Issues:
- When interacting with context menu on gamepad, the camera view will move if you don't let go before the menu disappears
- Some tooltips are hard to use without additional behaviors (e.g. LogiX tooltip)
- Currently key bindings are tenative and will likely be tweaked as more behaviors are added

Hello everyone! I wanted to do a bunch more work on the new desktop before releasing this build yet, but there's an important bugfix that heavily improved the cloud API reliability (this should fix sync errors and other problems due to interminnent network problems) as well as some important improvements for the face tracking!

The heuristics got expanded and improved, adding support for more avatars. Autodesk Character Creator avatars should now get setup with lip tracking (not 100% support either, but a bunch of blendshapes will be auto-mapped). Also there's now a mechanism where the voice data supresses certain blendshapes to avoid overdriving the mouth when talking with the Vive Facial Tracker.

There's a bunch of underlying work on the interaction system for the new desktop mode as well. Not that much visible yet, but touchscreen now has official initial support! Still more work to do, but you can play with it a bit if you'd like! I've got more coming soon!

- Added Timestamp to AvatarRawEyeData, which is relative time in seconds of the current snapshot of the eye tracking data (requested by @SHFR_H)
-- This allows calculating things like the current angular velocity of the eye movement with much better accuracy
- Added missing "Squeeze" and "Frown" properties to AvatarRawEyeData (based on report by @Ryuvi | Technical Artist)
-- Note that it's strongly recommended to use EyeManager, rather than this component unless you're building specialized application, as EyeManager is significantly more efficient and modular

- Added "Tongue Up Left", "Tongue Up Right", "Tongue Down Left" and "Tongue Down Right" expressions to the AvatarExpressionDriver (based on request by @umbran)
-- These drive based on the combination of both directions. Typically used for corrective deformation blendshapes
-- The heuristics has been updated as well to auto-assign those blendshapes (you will need to re-run the heuristics to assign missing blendshapes on already setup models)
- Added SmileClosed(Left/Right) expression to AvatarExpression driver
-- This will drive a smile that's only active when the lips are covering the teeth and will go to 0 when the teeth are visible
-- This can be used when there are separate smiling blendshapes for when the teeth are visible and when not
- Added VolumeSource and SilenceSource to AvatarExpressionDriver, which allow supressing the face expressions when the user is talking
-- VolumeSource is checked first, then SilenceSource (inverse of the first) when calculating supression
-- Each Expression now has "VolumeSupressionStrength", indicating how much is given expression supressed by volume (1.0 full supression, 0.0 no supression)
-- Existing instances are auto-initialized using VisemeAnalyzer on the avatar default supression weights are setup
-- By default, only jaw open and lips raising and pouting expressions are supressed. This should prevent the mouth opening too much when talking

- Added initial multi-touch support - the new desktop mode now properly handles touch screen inputs and multiple pointers for interacting with the UI
-- While touch inputs are active, the mouse input is supressed (this is to avoid double interactions due to Windows simulating mouse inputs from the primary touch)
-- Note that it doesn't go through the laser interaction system yet, there's more work that needs to be done on this
- You can now adjust the FOV (field of view) of the new desktop mode in the settings

- The FOV increases slightly in new desktop mode when running with the physical locomotion
- Extended the heuristics of AvatarExpressionDriver to provide better support for automatically assigning Autodesk Character Creator models
-- Note that you might need to tweak strengh of some of the blendshapes, like smiling
- Improved blendshape/bone name splitting to support naming conventions that start with a chirality letter (e.g. "RlipDown", which would previously split to "Rlip" and "Down")
-- This heaivly improves auto-detection of blendshapes on the Autodesk Character Creator models
- AvatarExpressionDriver no longer needs "shape" name for the "Ape Shape" (Jaw Down) name heuristic (based on model by @GearBell)
-- E.g. "mouth_ape" is now sufficient for the blendshape to be mapped, instead of needing "mouth_ape_shape"
- Preset names when starting worlds on the headless are not case-insensitive (based on report by @Epsilion)

- Merged Korean locale additions by @MirPASEC
- Merged Russian locale additions and tweaks by @Shadow Panther [RU/EN, UTC+3]
- Merged Japanese, Esperanto and Chinese locale additions by @Melnus
- Merged Czech locale additions by @rampa_3 (UTC +1, DST UTC +2)

- Fixed Neos not retrying Cloud API requests when the server returns 429 or 500 codes
-- This should greatly increase the robustness of any cloud based functionality under load or interminnent network errors, fixing cases where different processes would get stuck
-- This should also significantly reduce number of sync errors due to network issues (e.g. connection briefly dropping) as reported by @PlasmaRaven, @Elektrospy, @Honeypotbutt and @Shifty | Quality Control Lead
- Added a mechanism to manually correct Patreon support for accounts where the amount of support stopped reporting correctly from Patreon API
-- If your Patreon account got messed up and you're no longer receiving rewards, even though the charges are going through, let us know and we'll update your account manually
- Fixed GrinLeft using the right side of the upper lip to modulate
- Fixed invite and restart headless commands throwing exceptions when no world is focused (reported by @Glitch)
- Fixed non-host users showing "Hidden Contacts Only World" as "Contacts Only World" (reported by @Shifty | Quality Control Lead and @Cyro)
- Fixed Neos adding invalid entry to the configuration file when a world is started with invalid template and then configuration is saved (reported by @Epsilion)

Hello everyone!

We have some exciting news this week! HTC has released the VIVE Facial Tracker to general public, giving everyone access to full realtime facial tracking inside of VR. Neos has had native support for this hardware for a while, but with the release we have added an extra level of polish and automated the avatar setup to make it easy to get started.

If you haven't seen it in action, check out the video below. Combined with eye tracking, this hardware offers an unprecedented level of natural expression on avatars (huge thanks to Rezillo Ryker and GearBell for providing us with the most expressive avatars for demonstration) and allows scripting custom interactions that are triggered purely with your facial movements, like flying by blowing up your cheeks or activating fire particles on your head when baring your teeth.

To learn how to setup an avatar, whether a brand new one or existing one and even do some basic scripting, you can watch our tutorial video as well. It is split into several segments, so you can only watch the parts you're interested in!

And if you're looking to just give the face tracker a try, you can try it out on the new Neos Facebot, a fully free avatar by GearBell with full support for the face tracking working out of the box. You can find it in the “Face Tracking FaceBot Avatar World”.

We can't wait to see what you'll do with this new tech and what kind of amazing avatars and interactions you'll build!

In others news, we have made some progress on the desktop mode as well, adding support for proper aiming (a necessary step towards tools), control over FOV and even multi-touch support! You can read more about what we're working on below.

And also last, but not least, we have just crossed 1000 supporters and 15K on Patreon! We're overwhelmed with this level of support for this project. Without you, we wouldn't be able to keep working on it every day and remain independent, keeping the vision of the metaverse ours and its goals on providing the ultimate creative freedom. Thank you again everyone!



Last week, HTC released the new VIVE Facial Tracker add-on for the Vive Pro headsets to the general public, providing a real time tracking of the lips, cheeks, jaw and the tongue. Combined with the Vive Pro Eye headset, this gives you a full real time facial tracking while using VR, giving your avatar an unprecedented level of expressiveness.

Neos has supported eye tracking for almost 2 years at this point and face tracking since we got access to the devkit last year. With the release of this hardware to the public, we have applied an extra level of polish and functionality to make the use much easier.

We’re happy that this technology is now available to you, our community, as it brings the level of avatar fidelity in Neos to a whole next level, increasing the immersion for both you and other users, as your natural expressions, both voluntary and involuntary, now transfer to your virtual representation.

To demonstrate some of the possibilities, we have showcased the tracker with several different avatars (huge thanks to {LINK REMOVED}RezilloRyker and GearBell for providing us with highly expressive avatars to showcase), as well as some examples of basic scripting - triggering particles and special effects on the avatar with only facial movements and even flying by puffing up your cheeks.



To make full use of the facial tracker, you need an avatar that has the necessary blendshapes (face deformations) to visualize your facial movements. At the moment, there aren’t many avatars with a full support, but GearBell, one of our prominent community avatar creators, has created the Neos Facebot, a completely free avatar with a full set of necessary blendshapes.



If you’d like to give it a try, you can find it in the Inventory by going to Neos Essentials -> Avatars -> Face Tracking Ready. Alternatively check out the “Face Tracking FaceBot Avatar World” in the Worlds tab, where you can equip the new face bot and take it for a spin.

The avatar has some cool features built-in as well, including customization UI as well as jets and grappling hooks to play with, to showcase some of the cool avatar interactions you can have in Neos.

A huge thanks to GearBell for building this avatar to showcase this new technology!

When setting up a new avatar, you can now find a new option called “Setup Face Tracking” in the avatar creator. By checking this option, Neos will use heuristics to automatically map any blendshapes on the avatar to the tracking data coming from the face tracker.

The success rate and tracking coverage will depend heavily on the avatar. For best results, we recommend adding all the blendshapes from the sample models by HTC and following the same naming convention, to ensure that you get full use of the face tracker and maximum fidelity.

However Neos will perform even partial mapping, but some of the face tracking features will be missing with the particular avatar. For example we have added the Ready Player Me avatars to the heuristics, making them work with the new face tracker, but some of the face shapes are missing, such as the tongue movement.

We have recorded the following tutorial which showcases how to setup a brand new avatar, how to add support to existing avatars, how to customize the weights (strength) and even how to script custom behaviors based on your facial movements.



As part of the polish, we have also added a few of the missing blendshapes to the face tracking and eye tracking, such as tongue roll, tongue movement (left, right, up and down), eye squeeze and eye frown (currently doesn’t seem to be tracked with Vive Pro Eye however) and corrective tongue shapes.

We have heavily expanded the list of supported expressions on the AvatarExpressionDriver component, which serves as the primary way to drive the blendshapes on the avatar from the face tracking and added automatic assignment using heuristics (the same used by the avatar creator talked about above) to ease the setup.

Currently the heuristics are aware of the HTC sample models, Ready Player Me avatars and some general common face shapes. We’re working on adding a few more conventions, like the Autodesk Character Generator to work out of the box. If you have an avatar source that doesn’t work well with the heuristics and needs manual setup, please let us know on our GitHub with a sample model/naming convention!

The component now also allows you to tune the strength of the blendshapes as well, in case they’re too strong or weak. By setting the value to a really large number you can also channel the Garry’s Mod spirit.

We have built Neos to offer a huge amount of freedom for expression and creativity and supporting cutting edge hardware like this is part of that. Our goal is to always design systems in a highly future proof way to make sure we can adopt new hardware as it comes and make it available to you with as little effort as possible.

The face tracking is part of this as well and we’re planning to support more solutions as they come, including webcam based face tracking for the desktop mode that’s currently in development, without requiring any changes (or at least any significant ones) on your end and instead exposing them through the same set of components, like the EyeManager and AvatarExpressionDriver.

This is just the beginning for face tracking in VR and we can’t wait to see where it leads and what amazing stuff you’ll build with it yourself!


On our last livestream, we have showcased the face tracking with the VIVE Facial Tracker and the Vive Pro Eye headset

While a lot of our effort this week has been focused around the release of the face tracking, we have made some important progress on the official desktop mode as well. Currently our focus is on building out interaction systems, allowing the use of context menus and tools from the desktop mode.



This has a few major prerequisites, most of them that have been implemented already. We have expanded on the hand posing system, which now uses the actual avatar’s laser offset and is able to aim the hand exactly at a particular spot in the world, making sure the laser goes in a straight line.

https://www.youtube.com/watch?v=t02AyebR_gc

This part is crucial for building the tool interactions, as it will ensure that the simulated hand aims the tool at the exact point that your screen cursor is pointing at, whether it’s in first person or third person mode. Other interactions will be built on top of this as well, like physical grabbing or placing the hand at a nearby target.

Another important piece is extending the pointer interaction system to allow for a free-form cursor, allowing you to interact with the world and items within while freely moving the cursor around. To enable this, we have extended multiple systems. One of the benefits is that the desktop mode now has explicit control over the field of view (this is necessary to be able to calculate where in the world the laser hits).

https://www.youtube.com/watch?v=FAxGIBrYnPM

The second one is that Neos now has support for multi touch interfaces! Any existing UI can now be interacted with using a touchscreen. This will be particularly useful on mobile phones and tablets, providing a natural way to interact on those devices, although it still needs more work in this area, such as on screen controls for movement.

https://www.youtube.com/watch?v=7CRCwSWCX60

One of our longer ongoing projects is transitioning the renderer from a gamma color space to linear and providing more explicit control over color spaces for textures and colors in Neos. Linear color space will provide more accurate and consistent lighting, while managing the color space for the assets and parameters will ensure that your content looks the same (or as close as possible) as in the software it was authored in.

There is much to do for this transition, from the asset variant system, data model and the renderer, as well as making sure that existing content doesn’t break or look significantly different. We showed some early screenshots of the lighting and we’ll likely have more in the upcoming weeks as we progress.




The Metaverse Training Center is continuing development as well, with the different rooms getting filled out with content and polish. Here are some latest screenshots of the Streamer room and the Creation Plaza:




Creator Jam is back again! This week we focus on the four elements earth, wind, fire, and water. There were many Fifth Elements references abound, but we had some nice creations in this jam, so stop by and check out what people have made. Thanks everyone!


In this zone created by the keywords above, you feel like someone’s always watching you. Inspired by the .hack franchise, we have a fan recreation of a notable map from the franchise. Come here and listen to a prayer, or check out the stained glass directly in VR. It’s quite a lovely map and I appreciate all the effort that’s gone into it! Thanks Storm!


This wonderful sculpture of a tooltip brought to life by Beaned, showcases the magnificent beauty that is the Developer Tooltip. A tool made to make you a literal god in the Metaverse comes to life as it animates itself. Be wary that the Developer Tooltip doesn’t look for you for disrespecting it. (Yes this is a JoJo Reference) Thanks Beaned!


While it’s been in development, the VR Workout Club has been developing exercise machines in VR! Currently they only have a “Squatatron” (name pending), but it really shows that VR + Fitness can really go hand in hand! I’ve participated myself and it’s definitely a good workout! You really can feel the burn! If you wanna join try contacting one of them, anyone can join! Squat Away!


--------------------------------------------

Anyway that's all for this week, we hope it was an exciting one and that you'll have a lot of fun experimenting with the new hardware in the upcoming weeks as well. We'll continue bringing you more improvements over the upcoming days and weeks, as well as focusing on getting the desktop mode to basic feature parity, so we can perform an official swap.

You can keep an eye on the development on our official Discord in the #devlog channel and on any releases and update notes in #neos-updates.

See you next week!

Hello everyone! With the release of the Vive Facial Tracker, here's a bunch more eye and lip/mouth tracking improvements, to make the setup of avatars easier and more automated.

Heuristics of the blendshape matching were improved significantly and there's now a checkbox in the Avatar Creator to automatically set it up for face tracking. It's off by default, as it can mess up avatars if they're not setup for it, but feel free to experiment and see how well it works. You'll get best results following HTC's naming scheme (tutorial & samples coming soon).

There are some important improvements for the new desktop mode as well, notably with hand posing, which is now exact and capable of aiming the laser perfectly straight at the target. This is an improtant pre-requisite for implementing tools, as it will allow the system to aim the tool exactly at the point where the screen cursor is pointing.

A bunch of other improvements, tweaks and additions too, the VIVE Hand Tracking SDK was updated too (they seem to have made quite good improvements on the tracking quality!) and a few new blendshapes for the eye tracking that Neos didn't register before.

More to come soon!

- Added "Setup Face Tracking" to the avatar creator, which will scan the blendshapes on the avatar and attempt to setup face tracking
-- This can be used to easily setup avatars for the Vive Facial Tracker
-- Note that success rate will depend on the available blendshapes, Neos will use name heuristics to try to find best matches. You can get best results by following the naming convention from the HTC sample model

- Added Squeeze and Frown eye tracking parameters (based on feedback from @Reactant)
-- Appropriate fields were added to EyeManager and EyeLinearDriver
-- Squeeze indicates how tightly is the eye closed
-- Frown indicates the eye frowning (note that from my testing this doesn't seem to be currently tracked with Vive Pro Eye)
- Added StrengthMultiplier to AvatarExpressionDriver, which allows applying a global scale to the driven targets/blendsdhapes
- Added "Show Laser in Desktop Mode" setting which will display the laser in the new desktop mode when pointing at interactable objects
-- I mostly use this for debugging, but you can use this if you prefer for immersion

- Heavily improved hand posing in desktop mode when grabbing items and interacting with them
-- The actual position/direction of the laser is now respected, posing the hand exactly so the laser is straight when at rest and hand is properly offset for different avatars
--- This is an important mechanism for tool support, as it allows to pose the hand to aim at exactly specific point in the world
-- When the target point is near face, the hand is pushed to the side and down to avoid it from intersecting/obscuring the face/viewpoint
-- The hand interaction transition now tracks velocity, preventing instant snapping of movement direction and has tweaked velocities/smoothing
- The cursor reticle now fades away after 2 seconds of inactivity in the desktop mode

- Sliding items all the way towards the face will no longer equip/physically grab in then desktop mode
- Improved internal moderation tools to allow for more finegrained control (based on feedback by the moderation team)
- Userspace laser in desktop mode now correctly overrides the world-space laser, rather than having both activate at the same time
- Holding items in desktop mode no longer uses the simulated hand's twist to rotate the item, to prevent unwanted rotations when moving the item around
-- Note that some partial rotation might still be transferred when initially grabbing
- Improved Avatar Expression driver blendshape assignment heuristics
-- The heuristics now detects more keywords, improving face tracking support for Ready Player Me models and others
-- The assignment system now also has a face target filtering, ignoring ambigous blendshapes if non-ambigious are present (e.g. "Mouth Smile" vs just "Smile") and preventing multiple similar blendshapes from being all assigned, causing over-driving (e.g. having both "Mouth Open" and "Jaw Open")
- EyeManager parameters now use sliders to represent values within the 0...1 range
- Increased Stiffness on DynamicBoneChain decimal points from 2 to 4 (based on feedback by @H3BO3)

- Updated VIVE Hand Tracking SDK to 0.10.0 (from 0.9.3)
- Updated VIVE SR Anipal SDK (for eye and face tracking) to 1.3.2.0 (from 1.3.1.1)

- Merged Czech locale additions and tweaks by @rampa_3 (UTC +1, DST UTC +2)
- Merged Korean locale tweaks by @MirPASEC

- Fixed Interaction Laser raycast portals not working properly when the laser goes through another object that's ignored, causing UI to be unusable
-- This fixes cases of UI not being interactable in desktop mode when using portal raycast and the initial interaction hit hitting the user's avatar head
-- This also fixes cases of UI not being interactable when laser passes through canvas that let's the hit pass through if it doesn't hit anything (e.g. dash not being usable in VR when laser passes through the notifications canvas)
- Fixed grabbing a VRIKAvatar causing the neck and hips position to be taken from the held avatar, even though it's not equipped
- Fixed Cameras rendering the overlay layer
-- This should fix the mirror facet from rendering the dash while the desktop mode is enabled (re-reported by @epicEaston197 and @Khosumi)
- Fixed DictionaryList getting into invalid state when an operation during enumeration results in a single remaining item
-- This fixes various random bugs and potential corruptions due to using a list that's been returned back to memory pool
- Fixed out of range inputs for the maxUsers command on headless throwing an exception (reported by @Glitch)
- Fixed RecordSyncStatus not reporting sync error when the error is anything other than Out Of Space (based on report by @PlasmaRaven, @Elektrospy, @Honeypotbutt and @Shifty | Quality Control Lead)

Hey everyone, just a small patch to fixup a few issues with the laser reticle being too big in some cases, so it doesn't have to wait on other stuff.

Build is compatible with previous one, no need to update immediatelly if you don't experience issues.

- Virtual Keyboard will no longer activate when using the new desktop mode and clicking into a text field
- Tweaked laser reticle scaling logic in VR so it's non-linear, preventing the cursor from becoming too small when really close and scaling up slower when in distance

- Merged Japanese, Esperanto and Chinese locale updates by @Melnus
- Merged Russian locale update by @Shadow Panther [RU/EN, UTC+3]

- Fixed cursor reticle size not compensating for user's scale (reported by @Cyro, @Delphox, @Sox and others)

Hello and welcome to another weekly update!

This week we have some more important progress to share on the new desktop mode, which now has core interactions implemented! You can click, grab, interact with UI's and scroll with your mouse after you switch to the new desktop mode. The system will also pose your avatar's hand to make it look more natural.

There's still a lot of interactions and polish missing though - you can't use tools or context menus and the simulated motions are more robotic looking. Our focus has been on designing and building the underlying mechanisms, so there's a lot more to come before the new desktop mode is well rounded. For the time being it's still hidden behind the F8 key and fully usable only when Neos is launched in VR, but if you like to play with it in the current state you can!

Thanks to getting some development momentum, we have also addressed some of the highly voted GitHub/Patreon issues, focusing on ones that were easier to do to start. For example the dynamic bones scaling incorrectly after exiting an avatar anchor is no longer an issue.

Some of the work this week has also been more in the background. We have reorganized both our private and public GitHub repositories and published some missing ones. They're all now moved to the Neos-Metaverse organization account instead of personal Frooxius one.

This is to make the development process more robust with cloud automation, making collaboration with more developers easier and giving the repositories more official status. If you use the NeosPublic or NeosLocale repositories, make sure to update your links to the new ones!



If you missed our regular Friday livestream, you can watch the recording here. We did another usual session of Q&A, answering all your questions about Neos and showcased some of the progress on the desktop mode. This was before the interaction system has been implemented yet though, so for that one we recommend checking out the dedicated section below!


We have implemented the core of the new interaction system for the new desktop mode, giving you the ability to click, interact with world UI’s, grab and scroll with your mouse wheel. This new system is built on top of Neos’ existing interaction systems that have been used for VR.

Instead of rebuilding the interactions, the desktop mode simulates inputs that a VR user would and feeds the system interaction targets to make the inputs easy to use with keyboard and mouse or gamepad.



Thanks for this, the interactions will have consistency for both VR and desktop, with most systems in Neos not needing to distinguish in which mode the user is currently using. This simplifies development both on our side and for any content builders. You can also notice that the interaction laser cursor you see in VR is the same one that's now shown in desktop and changes based on the context the same way!

Not all interactions are implemented yet so far. So far you can click things (e.g. UI’s), grab objects and scroll UI’s with the mouse wheel. Using tools or context menus isn’t supported yet, but is coming soon.

Because of that, desktop mode is still considered in heavy development and is hidden behind the F8 key, but with this step brings even more usability to it. Our focus has been on building the core interaction mechanisms and systems that power it.


Some latest development notes on the interaction system

One of those is also a hand posing system. When you’re on a desktop and you click or grab things, the avatar hand will be posed automatically to make the interaction look more natural. Currently it’s very rudimentary and robotic looking, but once we focus on the layer of polish we’ll make it look more natural.

One of the new additions to the desktop mode is also the ability to crouch, giving you more flexibility and improvements to bindings for both mouse & keyboard and gamepads. Gamepad should now provide better precision for camera look, making it easier to do fine motions and can be used to turn in VR as well.

We have added bindings for avatar anchors as well, so any vehicles and other contraptions using the primary and secondary axes can now be used with keyboard & mouse and gamepad as well.

Other issues were fixed as well, for example the VR eye tracking is now ignored when in desktop mode, preventing the eyes from becoming derpy or the VR -> Desktop transition is now skipped when opening/joining a new world while already in desktop.

With desktop gaining some more momentum, we have put some time to some of the highly voted GitHub/Patreon issues. Our first pass focused on ones that are easier and faster to implement/fix.

The problem with dynamic bones scaling incorrectly after the user enters and exits an avatar anchor has been finally fixed. The LogiX node for baking meshes has also been expanded to give explicit controls over adding Colliders and Grabbable components to the baked mesh.

For a full list of new features, tweaks and bugfixes, you can always check out our Discord or the Steam patch notes.

Last week we have spent some time reorganizing our internal and public code repositories, to ease the onboarding process for new developers, prepare for a cloud automation of the build process and make things look a fair bit more official.

As some of you have probably noticed the public repositories (e.g. NeosPublic and NeosLocale) have moved from Frooxius account to Neos-Metaverse, along with any of our own open source libraries and forks that we use to build Neos.

We have also published a few libraries and forks that previously weren’t on GitHub at all or were private, for example our QuantityX library that provides unit conversions and formatting (it’s also historically the first library ever written for Neos).

While this doesn’t change much for you in the short term, this is an important step as we grow our team and development process, allowing us to better iterate on various Neos’ dependencies, making the development process more robust and improving collaboration.

While the interactions for the MTC avatar lobby are still being built out, we're already putting together models for another of the environments - the creator plaza. This one will serve as a tutorial / sandbox for many of the building tools inside of Neos VR. Here are some early previews of what it looks like so far, but please note that a lot can still change with them before they reach public.




Greeting and Salutations everyone! It’s time of the day for that weekly update of that good old community content. This week we are focusing on some maps with some sweet atmosphere this week. We also catch up to some of the stuff the folks over at Creator Jam have been up to!

An immaculate map made by GAWAWA. Come into this compostorium as you hear the divine musings of the music ebb and flow through your bones. From what I’m aware this world was made as a new home world and seems pretty grand as a way to start your Neos sessions! So if you ever get a chance take the time to check out the amp and look at the awe inspiring view. Thanks GAWAWA for the map!



Get ready to rip your pants folks! There’s a new skate party in town! Skate Park made by Gearbell, goes into setting up a nice scene for people to throw out a skateboard provided in the map to skate around and make some sweet flips! So grab a board, grab some friends and go skating! Thanks for the map Gearbell!



A wonderful world between worlds, a place between the “Everything”. A bleak world that fills a nice balance of void and existence. This place acts as a purgatory between the worlds of NeosVR. You look under your feet and see the energy of the Metaverse ebbing and flowing through the glass that you stand upon. Thanks Lewis_Snow for making this wonderful map to meditate and exist between everything! The sounds are really on point too!



For this Creator Jam we had a wonderful theme of where folks made all kinds of contraptions with buttons! Some even model exotic button types as you can by something making an actual “Belly” Button. Memes and jokes and all kinds of neat button creations in here!



Watchout! There’s a lot to look at or be looked at in this Jam! Creator Jam 92 “Whatcha Looking At?” takes a stab at using the new feature in Neos where you can have your eyes, ears, or 3d person to have a different perspective for your avatar. This is what allows people to have decoupled heads in Neos! People get a little freaky in here!



In this wonderful land of MacGuffin, there are many things to find in this location! Maybe you can find the Temple of Cheese? Creator Jam 91 MacGuffin Land focuses on interest points of discovery or to find for people to interact with. There’s some interesting things to find here so hopefully you find some treasure!



------------------------------------------

Anyway this is all for this week! As usual, huge thanks for your support, whether it's on Patreon, Twitch, ICO or just by being part of the community and building awesome content! We'll have more news for you next week, so stay tuned!

Hello everyone, got another important update to the new desktop mode, you can now grab and click things! It will even pose the avatar hand to make it look more natural. Everything is in quite early state so it's a bit rough and raw, as the focus is more on designing the underlying interaction mechanisms and building them out, but more polish will come soon. Hopefully it should significantly improve the usability of the mode in its current state though!

Sorry there's only few of other things in this build, there's a few bugfixes and tweaks as well, but my main focus was on getting more of the desktop mode going and dealing with some background things.

- Added initial version of desktop interaction system, allowing for clicking and grabbing in the new desktop mode
-- Current bindings are as follows (note that these are pre-liminary and will likely change as we go):
--- Left Mouse button / Right gamepad trigger to click/touch
--- Right Mouse button / Right gamepad bumper to grab/hold
--- Delete key to destroy grabbed
--- Insert key to duplicate grabbed
-- The interaction laser is used as a base, with the laser portion hidden and the cursor centered at the screen
-- Please note that this is still in heavy development and a lot more behaviors & polish is coming

- Added initial version of hand interaction posing system, which will pose the avatar hands when interacting with items
-- This provides a visual feedback to other users when a desktop user is interacting (e.g. clicking or grabbing items)
-- Note that currently the posting is still quite rudimentary and will be expanded and polished in the future builds
- You can now use mouse scroll for the in-world interfaces by hovering over them in the new desktop mode
- Added locomotion turn bindings to gameapads (based on feedback of Iforgotwhosorry ;_;)
-- Those are only active while in VR, in desktop mode they're overriden by the camera look controls which turn the player implicitly

- Added exponentiation (default 4) to gamepad look sensitivity, to allow for better precision with fine movements
- Laser reticle now scales based on the distance from the view for the owner, to make sure it's still legible even when interacting with far away objects
- Screen inputs now update before everything else, ensuring that no behaviors (e.g. avatar IK) lag behind its movements a frame behind
- Added "(for 7 days)" to "Remember Me" option in the login dialog (based on feedback by @drecalen#2829)
-- This is to avoid confusion on users getting logged out after a week of inactivity
- Tweaked update order for laser, fixing cases where the laser would be slightly jittery even when stationary
- VR -> Desktop transition is now skipped if the user is already in desktop mode and starts/joins a new world

- Merged Korean locale addition by @MirPASEC
- Merged Japanese, Esperanto and Chinese locale additions & changes by @Melnus
- Merged Russian locale additions & tweaks by @Shadow Panther [RU/EN, UTC+3]
- Merged Czech locale additions & tweaks by @rampa_3 (UTC +1, DST UTC +2)

- Fixed nametags going semi-transparent when the user is in the desktop mode
- Headless will no longer list and allow focusing of the Userpace world, preventing operations from being executed on it (based on report by @Glitch)
- Fixed headless hanging when running commands with improperly quoted strings and reworked parsing logic to allow for inclusion of quotes in the argument itself (reported by @Glitch)
- LogixTip now uses the bendy laser hit when restoring broken input nodes, fixing cases where custom tooltips would block the raycast when refreshing broken LogiX nodes (reported by @Hayden, @Shifty | Quality Control Lead and @Epsilion)

- The actual laser offset on the hand is not currently respected, resulting in the laser being bent even at rest and the hand getting into the face when moving item too close
- You can't use context menus properly yet (they'll spawn in incorect location)
- If you equip a tool, desktop interactions might not work correctly
- If you bring item too close it'll get physically grabbed/equipped
- Many more interactions are still missing, please remember that the desktop mode is currently in heavy development