3dify

Table of Content

1. Introduction
2. Application Front End
3. Architecture
4. Application Deployement
5. Run the Application Locally
   1. Prerequisites
      1. Common
      2. Windows and MacOS
      3. Linux
   2. Launch
6. Getting Started with Development
   1. Prerequisites
   2. Launch
7. Where to apply changes
8. 3Dify Test Suite

Introduction

The 3Dify project aims to enable avatar creation using advanced AI and a modular software architecture. Such a modular architecture allows for flexible and scalable development, ensuring easy updates and improvements.

3dify allows users to create fully-animated 3D avatars by uploading a single picture of a face. Using AI, 3dify scans the face from the input photo, extracts its features and uses the MakeHuman avatar generation suite to create 3D animated avatars based on the facial features extracted from the 3dify software modules.

3dify consists of 2 web applications.

1. a single page web application which allows authenticated users to browse previously generated avatars, upload new pictures and modify them. Once the picture is set, a “customize” button runs the Web GL application for preview and customization.

1. a Web GL application, built with Unity, that creates the avatar and renders the fully-animated 3d model result for preview. The application provides user interface controls to customize the output avatar by changing facial features.

Example of a generated avatar:

Application front end

The front end provides users with a gallery of pictures they have uploaded to generate their avatars. The gallery is proposed as a grid of photos to make it look familiar to users accustomed to picture galleries installed on their smartphones. Alongside the gallery, users are provided with a box to upload their pictures through drag-and-drop and selection from their computer.

Consider that a user uploads a picture either way; the platform shows users the update’s progress to keep them informed about what is happening to avoid refreshes or other actions that could only worsen the user experience, even though loading a picture does not take much time. When the loading is complete, users can access the picture from the gallery, and by clicking on it, they can preview the picture and gain access to several features. Below the preview, they have an action bar with options such as zoom-in, zoom-out, flip horizontally and vertically, rotate in both directions and the most important one–the customization option, which is dedicated to avatar customization and rendering using the Unity-based front end.

The application does not support logging in yet, but it is already designed with the capabilities to do so, and this is why users can already see buttons for logging in and out. This functionality will be enabled in future versions using the MongoDB database.

The WebGL front-end, developed using the Unity game engine, allows users to preview an initial version of their avatar based on the image uploaded to the web application described above.

After initial facial feature inference and avatar generation, the application displays a high-fidelity rendering of the fully animated avatar. This avatar includes a mesh with attached materials and textures, as well as a skeleton for use in applications such as XR and video games.

If the user is not satisfied with the initial results, the application offers extensive customization of facial features, including the head, eyes, nose, hair, and other details, using the panel on the left.

Customization is done by adjusting position and size values using sliders or by selecting from graphical options (eyes, hair, etc.).

By pressing the Build button in the lower left corner, the user initiates the avatar generation pipeline. This process, which takes more than 10 seconds, sends the new face parameters to the backend services to generate a modified version of the avatar.

Architecture

The application front end architecture comprises a web application, a file store, and a NoSQL database.

1. The web application is built with the Next.js framework, which allows us to develop both the front-end and back-end using TypeScript. The front-end is designed as a Single-Page Application (SPA) and written in React, also taking advantage of the abstractions offered by Next.js. Another benefit of Next.js is the possibility to use Next.js API Routes to create a serverless back-end to optimize resource utilization.
2. MinIO is used as file store for persisting uploaded photos, generated avatars, etc.
3. Part of the back-end is implemented in the Python language, which handles the facial feature extraction and communication with the Makehuman daemon
4. For the database, the application leverages MongoDB capabilities for storing users’ information and more.
5. The WebGL application is built with the Unity game engine and the C# language and lets users customize and preview the avatar and download its 3D model.
6. The avatar generation is handled by a Makehuman process running in background.

Application deployment

The application consists of seven docker containers:

• 3dify-makehuman: The container that executes a customized version of MakeHuman that permits to elaborate the sliders value extracted from a photo into a rendered 3D avatar.
• 3dify-unity: The container which starts a simple python HTTP web server which hosts the WebGL application for the avatar preview
• 3dify-python: Container containing the logic behind the conversion between facial landmarks and MakeHuman ’s parameters, as well as the logic connecting the application to MakeHuman for sending new sliders value and for exporting and downloading the final 3D model .FBX file.
• filestore: Container including MinIO, an object storage application compatible with the Amazon S3 API
• 3dify-next: Containers based on this image start the web application for the avatar management front-end.
• mongo: Container including MongoDB, an open-source document-oriented database based on NoSQL.
• mongo-express: Extension that allows to connect to any (local or remote) MongoDB server without having to install Mongo Express locally.

The deployment of these containers is coordinated by the Docker Compose configuration file.

Run the Application Locally

Prerequisites

Common

In order to use the authentication process, it is necessary to obtain a Google Client ID and a Client Secret from a Google Cloud instance and putting them inside the Docker Compose file. For the retrieval of informations, you can follow a guide at the following link.

Alternatively, if the authentication module is not needed, you can switch to the main-no-auth branch and build the application with the Docker Compose from there.

Windows and MacOS

For executing on Windows and MacOS Systems it is necessary to install beforehand Docker Desktop at the following link.

If when opening Docker Desktop for the first time on Windows machine should appear an error mentioning WSL try to open the command prompt and type:

wsl --update

(DISCLAIMER: Currently on Apple Silicon Processors the software may present some slow down due to the translation layer from x64 to ARM.)

Linux

For executing on Linux Systems it is just necessary to install Docker Engine by following the guide for your distro at the following link. (ATTENTION Currently this version only work by using Docker Engine with sudo command and is not compatible with Docker Desktop for linux systems.)

Launch

Download the Docker Compose file at https://github.com/isislab-unisa/3dify/blob/main/docker-compose.yml for the version with authentication, otherwise download the Docker Compose file here.

Launch all the containers required to run the application:

docker compose up -d

Stop all the containers of the application:

docker compose down

If the application is correctly deployed, it can be run by default at the link http://localhost:3000/.

Getting Started with Development

Prerequisites

Follow the same instruction as specified above.

Launch

Get the code at the repo https://github.com/isislab-unisa/3dify/tree/main for the version with authentication, otherwise get the code from this repo.

Launch all the containers required to run the application in development mode:

docker compose -f dev.docker-compose.yml up -d

Monitor the application while developing to see your changes reflecting automatically:

docker compose -f dev.docker-compose.yml watch

Stop all the containers of the application:

docker compose -f dev.docker-compose.yml down

If the application is correctly deployed, it can be run by default at the link http://localhost:3000/.

Where to Apply Changes?

• /app/api: in this folder you will find the code for the serverless APIs that power the application back end.
  • genderAge: which estimates the gender and the age of a person given a picture of the face.
  • photos, uploadPhotos: that lets the application read and write image files on the MinIO storage.

• /app/components: in this folder you will find the code for the UI elements of the application front end, such as the photos gallery.

• /app/pythonServices: in this folder you will find the code for the python-based back end.
  • scanFace: extracts the 478 landmarks that map the face of the input face portrayed in the image
  • extractFeatures: the outputs of genderAge and scanFace are processed to calculate facial parameters in terms of sizes and distances of all parts of the face (head, eyes, nose, mouth,…). Such parameters are numerical normalized in the [-1, 1] interval or a choice in an enumeration
  • applyAndDownload: communicates with the makehuman daemon which will in turn generate an avatar based on a series of facial parameters given in input

3Dify Test Suite

Prerequisites

In order to proceed with the execution of the tests suite, it is necessary to:

1. Install miniforge at the following link
2. Create a new environment named “makehuman” and activate it by typing

   conda create -n makehuman && conda activate makehuman
   

3. Install in the environment all the packages inside “tests/requirements.txt” by using

   pip install -r requirements.txt
   

4. Execute the test.bat or test.bash script

This site is open source. Improve this page.