Forensic Facial Reconstruction, the state of the art

As many of you know last week a team of the University of Leicester have publicly revealed to have discovered, in all likelihood, the tomb of Richard III. The results seem comforted by the analysis of mitochondrial DNA, while the discrepancy on the Y chromosome could be explained by a false paternity. The study was completed with a forensic facial reconstruction of the king, performed by the experts of the University of Dundee, led by Caroline Wilkinson, Professor of Craniofacial Identification.

Given the opportunity, I decided to publish here our state of the art on this particular field (forensic facial reconstruction applied to archeology), publishing the presentation that I gave during the study day in honor of Prof. Franco Ugo Rollo (Ascoli Piceno, November 26 2014).

You can see the presentation here below (better visualized at this link)...
 

... and here is a brief explanation of each slide:

SLIDE 1

A remember of Franco Ugo Rollo, professor at the Camerino University. It was not my fortune to know personally Prof. Rollo, but his name is surely well known also in my discipline (archeology).

SLIDE 2

"Digital faces: new technologies for the forensic facial reconstruction of the historical figures".
The presentation intend to be an overview of the digital methodologies of FFR with FLOSS, developed in the last two years on the blog ATOR with a spontaneous contribution of different authors.

SLIDE 3

The traditional work-flow involves several operations: 3D scanning the skull, preparing a replica, performing the anthropological analyses, placing the tissue depth markers, reconstructing the profile, modeling the muscles and skin, calibrating the model with the available sources and dressing it.

SLIDE 4

The same operations are necessary for the digital work-flow. Our main work has been to turn the traditional process into a digital one, using only FLOSS.

SLIDE 5

There are different technology to obtain a 3D digital copy of the original skull. The main two we are using are: SfM - IBM and X-ray CT.

SLIDE 6

IN 2009 Arc-Team perform the first test in applying SfM - IBM with FLOSS to Cultural Heritage, during its participation at the TOPOI excelent cluster of Berlin.

SLIDE 7

The test developed in a collaboration with the French researcher +Pierre Moulon (Université Paris - Est and Mikros Image; actually at Acute3D) to integrate SfM - IBM software in ArcheOS 4 (codename Caersar)

SLIDE 8

The first test (TOPOI Löwe) gave positive results

SLIDE 9

The process is mainly based on different photos with different orientations, computing the displacement of common points between images

SLIDE 10

To complete the 3D documentation of an object, the next step is the so-called mesh-editing, which can be performed in the software MeshLab (developed by the Visual Computing Lab at the ISTI - CNR of Pisa, Italy)

SLIDE 11

In order to validate the digital method of FFR, some unconventional procedures (derived from the hacker culture) have been adopted. With reverse engineering techniques, based on SfM, it has been possible to digitally replicate the process of past FFR projects and to compare the results.

SLIDE 12

The anthropological validation has been performed comparing the result of 3D models obtained with SfM - IBM and the relative results coming form 3D scan (the observed distortion remained in the range of 1 mm).

SLIDE 13

In several projects it is possible to work with DICOM data. In these cases the anthropological analysis is more accurate. (3D VS Voxel)

SLIDE 14

The main software we used for DICOM data is InVesalius, mainly developed at the Renato Archer Information of Technology Center, an institute of the Brazilian Ministry of Science and Technology.

SLIDE 15

"X-ray computed tomography (X-ray CT) is a technology that uses computer-processed X-rays to produce tomographic images (virtual 'slices') of specific areas of the scanned object, allowing the user to see inside without cutting." (Wikipedia)

SLIDE 16

Also in this case, the process was validated with unconventional procedures derived from hacker culture. With reverse engineering of CT videos it has been possible to rebuild DICOM data and the 3D model of different skulls, replicating FFR projects and comparing the results.

SLIDE 17

It is necessary to check and validate the protocol with a continuous methodological comparisonwith all the available resources. For this reason, we tried also the FFR of Henry the IV, a project in which Prof. Rollo was involved, rejecting (with other scholars) the attribution of the mummified head to the French king. Our test in this case is just an experiment, starting from low quality data, but it is a good example to show some benefits of digital FFR, like the possibility to quickly modify the reconstructed face (e.g. closing the mouth in order to perform superimposition with the death mask), an operation not so simple with tangible models.

SLIDE 18

Once obtained the 3D model, digital anthropological analyses do not differ from traditional ones.

SLIDE 19

In some cases, a virtual restoration of the model is necessary. The solution comes from symmetrical and boolean operations of 3D modeling software (Blender).

SLIDE 20

The whole process of 3D modeling is actually performed in the software Blender.

SLIDE 21

The first operation is to fix the 3D skull on the Frankfurt plane, which replicates the head position of a standing human figure.

SLIDE 22

Than tissue depth markers are placed. The software keeps automatically the correct normal of each marker.

SLIDE 23

In our works, for depth tissue markers, we use the tables of Degreef et alii (2006)

SLIDE 24

A second step is the profile reconstruction.

SLIDE 25

For nose shape we refer to G. Lebedinskaya method.

SLIDE 26

The validation of the method came mainly from the comparison between FFR models and the facial DICOM data of living people, a simple simple with digital techniques, using the software CloudCompaer. All this experiment were conducted ans blind test (the artist did not know the identity and the fisionomy of the people).

SLIDE 27

According to the blind test, main deviations were detected on the cheeks.

SLIDE 28

Like other 3D operations, muscles modeling has been performed in Blender.

SLIDE 29

The technique hes been continuously rationalized and optimize. For instance, once the main muscles are modeled with metaballs in Blender, the result can be reused in successive reconstructions through an anatomical deformation.

SLIDE 30

It is possible to reach more realistic results through specific modeling tools,
like the "sculpt mode" in Blender.

SLIDE 31

Also skin modeling is an operation to be performed in Blender

SLIDE 32

Again the technique has been optimized: In order to simplify and speed up the process, a neutral facial model has been  created.

SLIDE 33

The neutral model can be anatomically deformed on different skulls to meet gender and age dimorphism.

SLIDE 34

At the same time, the neutral model can be deformed to meet the anatomical criteria which determine the individual dimorphism.

SLIDE 35

After the reconstruction process, two main models are defined:  one with hair and one hairless.

SLIDE 36

Thanks to the latest developments of the software MakeHuman it is now possible to further simplify and speed up the technique. Our actual research is following this direction.

SLIDE 37

The first tests carried out in 2014 have yielded positive results, thanks to the new feature which loads base raster images. The software is also perfectly compatible with Blender.

SLIDE 38

A further development of the protocol will allow to obtain high quality forensic facial reconstructions, in less time, without the need to master the techniques of 3D modeling.

SLIDE 39

At the end of the FFR process, the final model is calibrated with historical, archaeological and medical sources.

SLIDE 40

In case of historical reconstructions, the model appearance (hairstyle and clothing) is calibrated depending on era and culture, while the physical characteristics (color of hair and eyes) are set basing on the ancestry.

SLIDE 41

The 3D printing technologies allow the materialization of the model with different levels of detail.

SLIDE 42

A case study: the forensic facial reconstruction of St. Anthony of Padua 

SLIDE 43

The 3D scan was carried out on the bronze cast performed by R. Cremesini in 1981.

SLIDE 44

The cast done by R. Cremesini is very important, because it derives from the temporary anatomical reconnection of the skull and the jaw, which were separated since the first survey of the tomb (1263). 

SLIDE 45 

3D scan has been performed with the SfM - IBM software of the archaeological GNU/Linux distribution ArcheOS.

SLIDE 46

The final model has been presented Tuesday, June 10 at the event "Scoprendo il volto di Antonio" at the Centro Culturale S. Gaetano in PAdua (Italy)  

SLIDE 47 - 50

Digital FFR allows to further define the details of the model to reach a more realistic result.

SLIDE 51

Thanks to the collaboration with the Centro de Tecnologia da Informação Renato Archer - CTI (Ministério da Ciência and Technology do Brasil) the model was printed in 3D.

SLIDE 52

One of the materialized models was repainted by the Brazilian Mari Bueno,
specialized in religious art.

SLIDE 53

Thank you for your attention!


 

Forensic facial reconstruction of a living individual using open-source software (blind test)

Studying alone is often a good solution when one cannot find support or has no understanding of something new and exciting, albeit not appealing to the general public.

Still, when it comes down to evolve and adapt scientific knowledge to the benefit of human beings, there is nothing better than having around people with the same goals, motivated to devote towards a better world, more accessible to those who have interest in that certain area of knowledge.

Earlier in 2012 I began my studies in the field of forensic facial reconstruction. Now, a year and a half later, over forty reconstructions have gone by, mostly of modern humans, some hominids and even a saber-teeth tiger.

Over that time, in the lectures I taught, in the e-mails I received or courses I offered, people often questioned me about the precision of the method, whether had I tested it in skulls of known people (living or not).

Graph representing the precision of a reconstruction (in millimeters) in relation to the skin of the volunteer, obtained by optical scanning. The blue areas represent areas where the face was reconstructed deeper than the real face, while the yellow areas represent regions in which the real face was deeper than the reconstructed mesh.

I had already done some experiments, but for technical reasons and in order to not disclose the identity of volunteers, I did not publish them. Instead, I was limited to showing the work of great artists such as Gerasimov from Russia, Caroline Wilkinson from England and Karen T. Taylor from USA.

Fortunately, a few days ago, research partner Dr. Paulo Miamoto sent me a scanned skull at my request, so I could test a newly developed technique to "wear" the skin over the virtual muscles. This skull, sent without much background on it, but with permission for reconstruction by its "owner", would be the first opportunity I had to show a case of facial reconstruction of a living person, exposing the degree of accuracy that such works may reach.

Development of the Work

A few days ago, I began to test a series of Blender modifiers, seeking an option that would allow me to "wear" the skin over a reconstruction in muscle stage. The goal was to make the process faster, and therefore more accessible to those who wish to replicate it, whether one is gifted with artistic skills or not.

I managed to find a solution with a modifier called Shrinkwrap (and a number of adaptations), as seen in the video above. The skull shown on the video is from another reconstruction in progress. It may seem almost imperceptible to a layman in forensic facial reconstruction, but it is a "blessing" for those who are just starting to work on virtual sculpture.

Back to the skull previously provided by Dr. Paulo Miamoto, it offered me the possibility to reconstruct a living person that was only known to him. He asked me for help with the configuration of the skull, since he would have to "assemble" the structure, because the CT was acquired by a Cone Beam tomograph.

Usually a cone beam CT captures only a portion of a skull due to a reduced field of view of the hardware. It is and equipment widely used for dental purposes and it is usually cheaper than a medical CT scanner.

An interesting fact in this story is that the whole process was done with open-source software. Initially, Dr. Miamoto opened the scans in InVesalius and filtered the part that corresponded to the bones. For this step he used a tutorial that I wrote, explaining the basic operation of InVesalius (translated from Portuguese): http://bit.ly/18mN6TR

Then he imported the three parts in MeshLab and aligned them in 3D space so that the face of the skull part stayed structure. All steps of this process were done thanks to the tutorials available at Mister P’s channel on Youtube: https://www.youtube.com/user/MrPMeshLabTutorials

After aligning the meshes the skull was exported as a .ply file and sent with the following anthropological data for the iorientation of the reconstruction:

- Gender: Male;

- Ancestry: miscegenated xanthoderm (of Japanese descent) and caucasian (white);

- Age: 20-30 years.

Upon receiving the skull I had to simplify the mesh, because the reconstructed CT had generated some areas with significant noise, inherent to the technique of image capture of Cone Beam CT scanners. Then I rebuilt the area of the skull that was missing by aligning it with another skull from my database, as recommended by the authors of the area. Thus, the work would be done more easily, with more spatial references.

With the skull cleaned and properly positioned in the Frankfurt plane, the virtual pegs used as reference for soft tissue depth were placed and sketches of the projections of the nose and face profile were done. As Asian and Native American individuals share physical anthropological traits that makes their skulls, a soft tissue depth table for the native indians from southwestern South America (Rhine, 1983) was used.

To speed up the process, a whole set of muscles, cartilage and glands was imported from another file. Obviously some changes needed to be donein order to fit it to the studied skul.

Gradually, one by one, the muscles were deformed and adapted to the skull.

At the end all the elements were positioned and contrary to what many people think, even with all the muscles of the face it is hard to get an idea of how the final work will look like, once finished.

For the configuration of the skin, the work followed the same method used for the muscles. A kind of general template is imported from another file.

And adapted until it fits the shape outlined by the profile sketch, muscles and soft tissue depth pegs.

It is possible to visualize the progressive shape transformation suffered by the skin mesh.

By placing the skin and "wearing it" over the muscles, I suspected the skull belonged to Dr. Miamoto. The shape of the chin and the side view highlighted some features that are evident in photographs (I do not know personally Dr. Miamoto). Upon questioning him, since in this field on cannot work with uncertainty, he told me “yes, it is his skull”.

Needless to say I was extremely pleased with the result.

Then it would be the time to test the quality of the reconstruction in relation to the face of skull "owner".

A test was done with a photograph, in which the reconstructed mesh was put over it and viewed from the same point of view. Note that the lips almost lined up with the 3D model.

Dr. Paulo then did the same process to filter the skin from the CT and sent it to me in another .ply another file. The file was aligned with the reconstruction, showing a rather large compatibility.

Finally a optical scan of the Dr. Paulo’s face (done apart from the CT scan) was aligned to the reconstructed  face. Note that again the line of the lips was quite compatible, as well as the nose breadth.

 

 The data of the reconstructed mesh and optical scanning mesh were loaded on CloudCompare and a 3D compatibility graphic was generated. A significant part of the reconstructed mesh differed only a few millimeters from the optical scanned mesh.

The part in blue, comprising the cheeks traditionally differs from scannings of the living individual because the soft tissue depth table used as reference was done on cadavers that may have undergone a slight change in its shape (due to dehydration and action of gravity upon its record).

This was an example of how a facial reconstruction done with open-source software can provide a rather satisfactory degree of compatibility with the living individual, provided it fulfills the current and already validated protocols.

The use of new technologies and specific tools in Blender 3D contribute to a satisfactory degree of compatibility of expression lines of the face, thus making the process faster and easier for those who wish to perform a reconstruction but often do not have an art training background.

The findings of this study are currently being structured as a scientific article. I hope to publish them in a peer-reviewed forensic journal, so that the technical aspects of using exclusively open-source software for forensic facial reconstruction can be adequately exposed and disseminated among those interested in this field.

Acknowledgements

To Dr. Paulo Miamoto for the continued partnership on several fronts of research involving open-source computer graphics to forensic science (and to translate this article for a decent English, thank you!)

To the Biotomo Imaging Clinic staff from Jundiaí-SP: Dr. Roberto Matai and Dr. Caio Bardi Matai for the CT scan of the reconstructed skull.

To the Laboratoř Morfologie a Forenzní Antropologie team, from Faculty of Sciences at Masaryk University in Brno, Czech Republic: Prof. Petra Urbanová, MSc. Mikoláš Jurda, MSc. Zuzana Kotulanová and BS. Tomáš Kopecký, for access to the collection of skeletal material of the Department of Anthropology, aid in research of photographic technique for photogrammetry purposes and optical scans.

To the Laboratório de Antropologia e Odontologia Forense (OFLAB-FOUSP) team, from Faculty of Dentistry at University of São Paulo: Prof. Rodolfo Francisco Haltenhoff Melani and MSc. Thiago Leite Beaini for supporting the works in Brazil.

To the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES):  for granting a scholarship for Abroad Doctoral Internship Program (PDSE).

A skeleton of Medicine's History Museum have his face revealed

Some days ago I was looking for some skull to reconstruct and present in my talk on Blender in Brazil. The talk will happen in the FISL14, one of the greatest free software conference in the world.

Fortunately I received an email from the Medicine's History Museum of Rio Grande do Sul (MUHM), that needed a forensic facial reconstrucion.

The nickname of the skeleton is Joaquim. He was a prisoner that died like a indigent in France in 1920. In 2006 he was donated to the museum by a family of doctors.

I ordered a CT-Scan and the people of the museum sent me not only the head, but all Joaquim's body.

So, I'll reconstruct all the body, but for now only the head was done.

To reconstruct the bones in 3D I used InVesalius, a CT-Scan reader open source. It was necessary export some files with different configurations, because the amount of data is huge.

Like I said, in this first part of Joaquim Project I'll reconstruct only the face. In the Meshlab I cleaned the noise of 3D reconstruction of CT-Scan.

The skull was not complete. To get the mandible I made a projection using Sassouni/Krogman method shown in Karen T. Taylor's book.

With the help of forensic dentist Dr. Paulo Miamoto, we get the range of Joaquim's age: 30-50.

The tissue depth markers was put.

So it was possible to sketch the profile of the face.

 

 The muscles was glued at the skull.

 

 Finally, the skin, the cloth and the hair was put.

I don't know if Joaquim really was born in France, but he appear a French man.

Thanks to:
Éverton Quevedo and Letícia Castro from MUHM.

A big hug and I see you in the next!

Paranthropus boisei - forensic facial reconstruction

In the first works I made involving forensic facial reconstruction, It was important to me modeling all from scratch. More than to model, I created all textures and illumination in each new work.

With the time, and with the experience, I noticed that some properties of that works repeated constantly.

Because this, I developed a methodology to make the reconstruction faster, both with humans as hominids.

In this post I'll show you how was the reconstruction of a Paranthropus boisei. The work, how ever, it have the help of the archaeologist Dr. Moacir Elias Santos. He took some excellent photos that was the base of the 3D scanning with PPT-GUI.

Using CT-Scans of a Pongo pygmaeus and a Pan troglodytes (chimp) how references, the muscles was modeled.

Because of the morphology, we decided to use a CT-Scan of a chimp how reference to be deformed and match it with the mesh of the P. boisei. We used InVesalius to reconstruct the CT-Scan in a 3D mesh.

While I deformed the skull, the skin got the appearance of a new hominid.

The resulting mesh was the reference of the final model.

Instead of modeling the P. boisei from scratch, I imported the mesh of an Australopithecus afarensis to be deformed and match it with the skin base deformed from a CT-Scan.

By editing the mesh was possible conform it with the skull and the muscles of the P. boisei.

The edition of the mesh in Blender Sculpt Mode was done with a digital tablet Bamboo by Wacom (CTL-470). Surprisingly it was not necessary install anyone driver on Ubuntu Linux.

To finish the work, I made the texturing and put the hair. The render was done with Cycles.

I hope you enjoyed.

A big hug!

PPT GUI and Blender, an interesting solution to forensics

Since I started to study forensic facial reconstruction, more or less one year ago, I had the opportunity to know a lot of interesting things and people that works in a couple of different fields, but with something related to forensics.

Currently me and Dr. Paulo Miamoto scan a skull using SfM to test the accuracy of the PPT GUI in relation to laser scanning.

This is very important because shows an interest of instituitions  related to forensics science relative to free software.

If is proved that the accuracy of SfM is compatible with the laser scan, it can be an important and revolutionary opportunity to take the accessible technology for a lot of parts of the world that never thought about use it before.

We hope this attempt be well succeded.

Acknowledgements to Mikolaš Jurda and Prof. Petra Urbanová, head of the Laboratory of Morphology and Forensic Anthropology of the Faculty of Sciences, Masaryk University (MUNI); Prof. Rodolfo Francisco Haltenhoff Melani, head of the Laboratory of Forensic Anthropology and Dentistry of the Faculty of Dentistry, University of São Paulo (OFLAB-FOUSP); and CAPES Foundation, Brasília - Brazil.

The Faces of Evolution - Exhibition of hominids forensic facial reconstructions

May 18th through July 31st, 2013 will happen in the big and beautiful Curitiba city in Brazil, the exhibition The Faces of Evolution.

Near the replicas of skull of hominids will have nine panels with forensic facial reconstructions.

The exhibition will happen now, but the history started here in ATOR some months ago.

The methodology developed to model the majority of the faces was used for the very first time in Alberto di Trento's reconstruction.

On that occasion the Arc-Team presented a paper at the event Giovani Antropology (Italy), where they described the methodology for forensic facial reconstruction using only free software.

Following the events, we initiated Project Taung. If before we had rebuilt a modern human this time we would rebuild a hominid remote, then the Taung Child.

Beyond the Arc-Team Group, has joined the project Prof. Dr. Nicola Carrara University of Padua.

We were able to reconstruct the face of the Taung child fastly, thanks to a methodology developed previously.

We had a largely positive feedback, both personalities linked to forensic facial reconstruction, as those related to 3D modeling, as the creator of  Blender, Mr. Ton Roosendaal.

Several media outlets broadcasted stories (see the comments) on the reconstruction of the Taung child, as the Jornal de Brasilia above.

After the successful reconstruction of hominid remote, we got in touch with the archaeologist Prof. Dr. Moacir Elias Santos of the Archaeological Museum of Ponta Grossa and Prof. Esp. Vivian Tedardi of Rosicrucian and Egyptian Museum in Brazil and we began a series of modeling which culminated in the exhibition the Faces of Evolution, described here.

The Australopithecus afarensis.

The Homo erectus pekinensis.

The Cro-Magnon man.

The Turkana Boy.

The Homo heildelbergensis.

The Homo neanderthalensis.

The Homo habilis.

And last but not least, the Homo floresiensis.


This was a brief description of the exhibition and its history. Now, we expect the event date arrives, while others look for fossils to be rebuilt.

A big hug!