Rhinoplasty planning with RhinOnBlender addon

Rhinoplasty planning steps: Measurements, sculpture, post guide, pre-post guide.

I always thought that plastic surgeons were professionals that only served the moneyed elite and who passed away from the treatments that benefit the users of the Brazilian public health system. When I was asked to teach computer graphics courses to these specialists, I had the opportunity to follow their work more closely, which, unlike what I assumed, involves a routine of scientific production, clinical care and a tremendous effort to offer the general population the possibility of solving a series of problems, ranging from a structure that leverages the self-esteem of this individual, to a procedure that returns a motor skill that was limited, either by accident or pathology.

By teaching classes and adjusting Blender parameters to the needs of plastic surgeons, I realized that activities could be performed much more effectively if the commands were grouped into scripts, and these, in turn, were attached to buttons and pre-structures defined, provided by the graphical interface itself.

The idea of creating RhinOnBlender came out after we made Cork on Blender (boolean addon) available and started working on OrtogOnBlender (orthognathic surgery planning). Although the latter is not available, for the moment to the general public, we are already using it for the planning of orthognathic surgeries, which deal with facial deformations. Similar situation has happened with the rhinoplasty addon. It was born from a process of simplifying commands to beginners and we are already using it for planning, yesterday we did one!

Automatic attenuation of the 3D mesh by photogrammetry - option implemented just now

The development of the tool is the result of a partnership with three specialists: Dr. Pablo Maricevich (Cir. Plastic), Dr. Rodrigo Dornelles (Cir. Plastic) and Dr. Everton da Rosa (Cir. Dentist). Two plastic surgeons and a dentist? That's right, although they look like different areas, the planning tools of one field and another are quite compatible, what we did was the joint development over the cases that we have worked on since the year 2014.

Experimental guide made from pre and post surgical mesh

The goal of this project is to offer an alternative based on freeware and free software. The video featured in this post uses a face scanned in 3D by photos (photogrammetry), there are  licenced, freeware, free software, offline and online options of this tool, such as the popular Photoscan, Recap 360, MVE and OpenMVG. But nothing prevents the specialist from importing a face scanned by laser scanner or structured light or even reconstructed from a CT scan. The point here is that there is a possibility to do a good job without having to pay for the tools, since we have free photogrammetry and Blender and its addons are also free!

Experimental guide from different viewpoints

Speaking of addons, I can not fail to mention the enormous help of two developers: Dalai Felinto who wrote Cork on Blender, a tool that boosts the work with boolean, essential when it comes to modeling for healthcare. Also the specialist in dental tools, Patrick Moore, better known in the cybernetic environment as Patmo141. The addons developed by him are geniuses and two of them are used in ours, Cut-Mesh, which lets you draw on a mesh and isolate the desired part and Object-Align, which as its name says, aligns objects, something that we don't have available in the standard Blender installation.

We are working hard. At each surgical analysis and surgical planning we see the need for repairs and increases in the code. Once the tool is more mature and fluid, we will make it available to anyone who has an interest. We hope this happens as soon as possible.

Until then!

Digital 3D Orthognathic Surgery with the OrtogOnBlender addon

Timelapse video of orthognatic planning steps

The first time I understood the concept of programming, it was in the year 2001, when I realized that a text with some codes was translated into a three-dimensional scene with some animated elements. I was astounded by it, as when I was three years old I realized that the water that was coming into the refrigerator freezer were those transparent pebbles my grandparents used in the juices they prepared for the family.

Since then I have always consumed some information about programming. Sometimes it would even cheer me up and write some codes, but the lack of practical use made me confine myself to just exercising my brain to understand some abstractions that involved that reality.

Many years and many books later, here I start teaching 3D computer graphics courses for the health area and realize that the organization of Blender graphic interface more confused than it helped the doctors in their didactic absorptions.

Faced with this difficulty on the part of the students, I motivated myself to create a methodology that would facilitate students' understanding in a field that proved to be quite challenging in the didactic context: orthognathic 3d digital surgery planning.

Automatically created deformation area

Unlike the other courses I had taught so far, orthognathic surgery, when taught in the "manual mode," where the student needs to understand detail by detail of commands, is almost impossible to apply to a short course. The problem is that it is precisely short courses that are the most sought after and not offering them translates into not generating income.

Together with Dr. Everton da Rosa, Dentist, orthognathic surgery specialist, I began to develop a series of small scripts that would help students automate one task or another. I did one, it worked, I cheered up... I did another, it worked out and another... I picked up a little and kept staggering and achieving, although not in the most elegant way, at least it was functional.

Jaw rotation

Blender uses Python for its scripts and this language is marvelous for anyone who wants to try to program, since the code is pretty clean. My excitement was growing to the point that I was fissured by the small challenges that were appearing.

Initially I intended to create a series of scripts, but realized that I could bundle everything into an addon, which would be installable in Blender. When programming and working with different objects and modes, I was intuitively understanding how everything happened, to the point of thinking of a solution, writing without referencing and code working!

Automatic creation of vertex groups

It was what I needed to become a hermit cyber... locked in the bedroom, just thinking about how to solve the problems and expand the functionalities. In unlikely five days, though not finalized, I was able to mount an addon sufficiently functional to simplify and automate a series of steps that comprise the planning of orthognathic surgery.

Interestingly, as a good nerd who follows the best practices of self-taught effort, I did not ask anyone for anything, I did not post on forums or social media groups on programming, nor did I ask for help from friends who are masters of language. I found everything I needed in the documentation, videos on Youtube, tutorials on the internet and, pampering ... in templates provided by Blender himself! In short, I did not ask for direct help, but I got it from the work and goodwill of people who wanted to help and made excellent materials, I am very grateful to all!

Jaw after osteotomy (Cork on Blender)

I intend to make the addon available soon (as we did with Cork on Blender), after testing it with my students. It is not yet complete, since it lacks the part of the surgical splint (a sort of guide for fitting the teeth) and some other activities such as automatic parenting with bone armature.

Even so, I am very satisfied, mainly for having implemented the automatic process of creating areas of influence based on osteotomy. With it, the cut bones, when moved make the soft tissue (skin, fat, etc.) deform. You can not imagine the challenge of teaching this in the "manual mode." Luckily it all came down to clicking two buttons, which is not bad!

Deformation radiated by the skull

I'm so happy and motivated that the will it gives is to go around programming and automating everything, but I felt on the skin another situation that is inherent in this kind of knowledge ... a code needs to be elegant and be prepared for exceptions and mine does not pass nor close to it. The trend now is to lapse the code and try to make it decent, worthy of being shared, used and improved by the community and interested in using it.

That's it, now I'm going because I had an insight on how to solve a problem in one part of the code... a big hug! : D

"Cork on Blender" the addon that has saved lives is available to everyone!

Retrieval of low grade osteosarcoma with virtual planning of microsurgical reconstruction with fibula flaps. Credit: Bruno Aragão.

"First do, then show"... this has been the mantra of our team. Since August 2014, three years ago, great challenges have arisen when I was contacted by Dr. Everton da Rosa to teach him 3D computer graphics techniques so that he could use them in his virtual orthognathic surgery plans.

Orthognathic surgery is a technique that corrects facial deformities in adults. I myself would need this procedure, since my upper jaw is small and the mandible is projected. This makes me look not very pleasing to the eyes, but fortunately I breathe well and ... I'm married, what would be an accomplishment for an antisocial nerd ... imagine for a short, antisocial and ugly nerd!

Joke aside, many people have serious problems as a result of these deformations, which often puts the lives of these individuals at risk. When planning a surgery digitally, specialists such as Dr. da Rosa have the opportunity to approach the procedure without the risk of causing harm to the patient, and when performing the task effectively, will be very well prepared, since he did the studies several times before surgery.

Orthognathic surgery planning with Blender and Cork.

This whole digital process would work very well, if we did not have a problem on the way, this problem was Blender's native Boolean. It should be recognized that with each version the library of these calculations has become better, but when we speak of boolean for use in health sciences it presents problem at inopportune moments.

To solve this problem, still in 2014, I looked for a classic exit for those (like me) working with Linux and free software: research a project that would allow the robust calculation of Boolean. I ended up finding it in the tool Cork, one of the great works of the developer Gilbert Bernstein.

Examples of using Cork on Blender. Left: Reconstructed hull of Freddy's turtle, in the center: facial prosthesis (Credits: Mais Identidade and UNIP) and left: prosthesis of a beak.

I was delighted with the results and proceeded with the tests, although these depended on exporting the files for the calculations to be done externally. As the activity was not very practical, came into the picture the one who made use of Cork within our beloved Blender, the expert in Python Script Dalai Felinto.

He masterfully created a code that turned a series of boring activities (export, command line, etc.) into a few grouped buttons on the upper left of 3D View. This was the trigger for a number of interesting projects.

From that moment, we started to do the surgery plans almost completely within Blender. We imported the reconstructed (skin and bone) surfaces from tomography and performed the planning, now without suffering from the part of the osteotomy, the one in which we cut the bones.

With Cork inside Blender, the osteotomy, which once drove us crazy, was now just another process, being performed not only efficiently but quickly.

In the course of the planning, other projects have touched our daily life, from the maintenance of cultural heritage, to the making of facial prostheses for humans and in the field of Veterinary Medicine.

Together with the Animal Avengers group and specialists from Italy, I created prostheses for no less than 11 animals! There were four toucans, an aracari, a dog, a goose, a crow, a turtle, a parrot, and a macaw. In addition to this area, I worked with Dr. Rodrigo Salazar with the manufacture of facial prostheses for cancer victims who had one eye removed.

Over time, a lot of people started looking for our teams so we could provide training in all of these areas and we would naturally install the addon on the students' computers. In the face of the great demand and security provided by it, we finally decided to share this tool with everyone and today we have the honor to announce it available openly and for free!

For those who want to download, just access the following link:

https://github.com/dfelinto/cork-on-blender

And follow the instructions.

You can compile Cork or download one of the versions available for the three operating systems: Windows, MacOSX or Linux.

I would like to thank all those who have helped us to develop this tool, either by learning to use it or by making ourselves available to be part of the research.

A big hug!

3D documentation of ancient millstones: preliminary tests

The traditional drawing of ancient millstones consists in a plan (eventually with shading to give a sense of three dimensions) and one or more cross sections of the object. This is not always easy because of the dimensions, weight and sometimes shape (mainly the Pre-Roman millstones are irregular and asymmetric) of this type of artefacts. Furthermore, the millstones are generally in museums or in storerooms: in these places, it is often difficult to move the objects or to have enough time for drawing quietly and checking well all the details. In short, drawing a millstone is not like drawing a sherd of pottery!
For these reasons, it could be useful applying a methodology based on the Structure from Motion (SfM) techniques in addition to the traditional drawing methods. In this post I’m going to present the preliminary results of a test aimed at the three-dimensional documentation of a fragment of an Iron Age millstone from Northern Italy (a so-called “Olynthus mill” or “Hopper rubber”).

The first step was the construction of a rectangular wooden frame made by 4 rods of different length (40, 60, 80 cm), so that it’s possible to build frames of different areas according to the dimensions of the millstone to be drawn. On the surface of the rods some cross marks equally spaced are signed: these marks will be use as reference points with known coordinates for the rectification of the3D point cloud, mesh and (eventually) texture (something like GCP, Ground Control Points).

Four bolts adjustable for height hold the frame together and allow to level it perfectly. Once the frame is ready, you need to enter the millstone into it, in such a way as to leave sufficient space between the stone and the rods for taking pictures.
Some recognizable markers should be placed in different points of the millstone: these are for aligning and merging the two point clouds that will be generated (see below). A simple solution is to use small spheres of coloured modelling clay visible in the point clouds.

At this stage, you start with the typical workflow of the SfM. You take an appropriate number of pictures of the upper surface first; then do the same to the lower surface, turning upside down the millstone inside the frame.
The pictures can be processed by the software you want. I used Regard3D / OpenMVG for generating two point clouds (one for the upper surface and the other for the lower) and CloudCompare for editing/cleaning the point clouds and for performing their rectification (thanks to the cross marks on the frame), alignment and merging (thanks to the coloured markers on the stone). CloudCompare and MeshLab have also been used for generating meshes and for computing other parameters, among which the measurements.

The final result is a point cloud and a mesh of the millstone.

 

Using MeshLab you could also obtain the texture of the object, but for my aims it’s enough a 3D model (point clouds or mesh) from which I can get a plan, some cross sections and all the measures I need. Thanks to these data, I can detail or check my handmade drawing or do it from scratch.

In conclusion, the usage of an homemade wooden frame makes easier and more precise the data acquisition for the SfM and make faster and more complete the documentation of this kind of artefacts. The method described leaves room for improvements and developments; it could become a “standard” documentation technique for the ancient millstones and for other archaeological objects with analogous drawing issues.

Thank’s to Alessandro Bezzi (Arc-Team).

Denis Francisci