Reassessing 2D photo to 3D model processing, using free tools

Previously, I wrote a blog post on using ReCap Photo to create a 3D model from a set of 2D pictures. Some of my friends that are interested in the topic let me know they would rather perform the process using open source (free) tools.

I researched several options and came upon:
Meshroom: a free and open source photogrammetry software from AliceVision. 

The tool appears to be very tunable with many knobs to turn at every level of the process. It also has a solid user interface that is relatively easy to use.

Unfortunately, detailed documentation about what all those knobs can do is a bit lacking, though there is a post with instructions on running through the process of using Meshroom, using a photoset available here. There is also a wiki about Meshroom available. There is also a good YouTube overview of Meshroom and a discussion of the process.

When you install Meshroom, place the installation folder on the top level of your hard drive. It seems to be overly sensitive about non-alphanumeric letters in the file structure (including spaces). Meshroom will only run on systems with a CUDA (nVidia) GPU. The rest of this post is a walkthrough of the process.

1. Once Meshroom is installed, you can drag your photos to the Images window on the left hand side of the screen.

Images window

Note that each of my images has a little red circle beside it. This indicates that Meshroom cannot determine the type of camera used to take the picture. It gets the camera information from the metadata inside the picture. If you really want to get serious about this, you can define the focal length… for the pictures and the tool will do a better job. The better the camera the better the results. I just used my cell phone.

• Once you have loaded up your pictures and are ready to begin, pull down the File menu and define the location you would like to have the files stored. I recommend you do not try to place them in a cloud-based folder, since that will really slow down the process. Also try to make the file path contain only alphanumeric letters.
• Once the folder location is defined, click the green Start button at the top of the screen.

Start button

There is a progress bar that runs along the top of the screen to show you how the process is progressing.

Progress bar

The yellow section indicates how far along the process has proceeded. If the program encounters an error, a red section will be displayed. You’ll need to dig into the logs to see what happens at that point –more on that later.

You can also see a progress indicator at the actual stage being executed:

DepthMap node

In this case, we are about 1/5^th of the way through the DepthMap step. This stage will take a very long time to complete, so expect the process to thrash your computer for possibly hours. By the way, don’t let your computer fall asleep until the process completes.

All the steps in the process are displayed in the Graph Editor window of the screen. These various processes pipe their output to each other.

Process steps display

For each of these nodes, you can click on them to see the log of what was being worked on, if a problem occurs. You also have a real-time log window that tracks the entire process.

As the process continues, the program will evaluate all the photos and acceptable/usable photos will be marked with a green checkmark in the Images window.

Images window

If any images are determined to be flawed, remove them, and start the process again.

As the process continues, you can see work in progress in the 3D viewer window:

3D viewer window

In this window, you can pan, zoom, and orbit the work in progress model using the mouse. More detail in the point cloud will appear as the process runs. The white boxes on the screen represent where the program thinks the cameras were located.

Once the process completes successfully, you will see a button titled Load Model.

Load Model button

Use this button to preview the results.

To work on the model that was generated, a folder is created where you told the program to store the model (way back at step 2). The folder called MeshroomCache.

MeshroomCache folder for in-process files

Within that folder will be a hierarchy of files, containing work-in-progress data that was transferred between the various stages of the recognition process:

Work in process file heirarchy

Though the various intermediate files may be useful, you are likely most interested in is the .obj, .mtl and texture file in the Texturing folder.  To access those files, you will need to go one level deeper in the Texturing folder, past a randomly named directory.

Texturing folder contents

The .obj The OBJ file format is a simple data-format that represents 3D geometry alone — namely, the position of each vertex, the UV position of each texture coordinate vertex, vertex normals, and the faces that make each polygon defined as a list of vertices, and texture vertices. This format can be imported into many (most) 3D modeling software.  This mesh will be high a high definition model. You can use a tool like Instant Meshes to view the results and simplify the model (here is a video explaining Instant Meshes use).

The results of my efforts were not quite as precise as the results from Recap Photo, but the process was still effective. I am sure if I experimented more, I could improve the results. Here is what the generated model looks like in Meshlab:

Load of initial model into Meshlab

There are over 1.2 million faces and 621,000 vertices. After running it through Instant Meshes, the model looked like:

Post simplification processing model in Meshlab

It now has 249,000 faces and 95,000 vertices. We lost the color (which we didn’t need anyway) and a much more simplified result was generated. We can now crop out the areas we don’t want and end up with a mesh we can work with.

3D model creation using my cell phone’s camera

Recently, I was in a discussion with some folks in our Technology Enthusiasts group about 3D model creation. I dabbled with this several years back, purchasing a Microsoft PC interface for the Xbox One camera. It worked, but only on fairly large objects (human size).

I wanted to be able to create models of smaller things, so I investigated Photogrammetry. This is the technique of taking multiple overlapping photographs and deriving measurements from them to create 3D models. The basic principle is like the way many digital cameras allow you to create a panoramic photograph by stitching multiple shots together.

I have been creating 3D printed buildings in HO gage for the model train group here, so I thought I would work on a model that size. My wife has a miniature Christmas village that has a cinema building, so I thought I would try and scan that in. It took 3 tries but I think I ended up with something viable. To get started you need to understand the process:

1. Place the model on a flat surface you can move around. I tried placing it in an isolated environment with a bland background (e.g., surrounded by cardboard) but that did not work as well as just taking many photographs as I moved around the box. In my case, I used a table on my porch, since it had good lighting and I could move around smoothly.
2. Take a series of pictures around the object and from various angles. In this case I took a straight on shot and one slightly elevated shot and moved around the object about 10 degrees each time. I did not care about the bottom of the model; I knew it was flat. My goal was to get 100 good shots (well-lit and not blurry) for the software to work with.
3. Once all the shots were done, I loaded them into Autodesk Recap Photo. There are many tools out there but since I use Fusion 360 and there is a hobbyist license, I settled on this one:
   • Start by loading the software and creating a new project.

Creating a project

• Load the images into the software, using the button on the left.

Loading pictures into the project

• Once all the pictures are loaded, and you have made sure you have only the good ones you want selected, click on the Create button at the bottom of the screen to start the process of creating a 3D model.

Creating a model

• This is a cloud-based process that runs on Autodesk managed servers. All the images need to be uploaded and take can take a while (3-5 minutes). Progress is shown in the icon at the bottom of the screen.

Uploading to the server

• Once the upload is complete, the job will be placed in a queue with everyone else wanting to create a 3D model and the waiting really begins. The 3 models I loaded took at least 4 hours to complete. I was not there watching to see the exact time they completed.

Waiting for the model to be created

• Once the job is complete, a small arrow will appear, allowing you to download the model to your PC. After you click download, a dialog will appear asking you where to store the model. A new folder will be created in that location with the model.

Downloading the model

• Once you have downloaded the model, a new model icon will appear in ReCap Photo in the My Computer section of the screen.

Opening a new model

• When you open the model, you should see a 3D rendering of the object. Note, there will likely be some anomalies. You can use the tools on the left of the screen to address many of these. This model can be rotated and manipulated at this point.

The 3D model

1.  You can export the model in numerous formats:

Exporting to a model other software can use

• In my case, I saved the model as a .STL file and loaded it into 3D builder to scale and clean up further. You will likely need to use a mesh editor to simplify the file before Fusion 360 can turn it into a solid.

The model loaded into Microsoft 3D builder

Hopefully, you will find this tutorial useful. There is also a great video on how to manually create 3D models from photographs out on YouTube using Fusion 360.

Time and the Corona virus

I recently had a few situations that made me think about an old Alan Parson’s Project song called Time. So far in my retired life, I have been busy with meetings, activities… so time was still divided up into quanta.

With the advent of the Corona virus, time turned into a flow. Units of measure blurred to the point where even the ‘Groundhog day’ perspective broke down. When a meeting that I should attend was defined (several times) I didn’t even recognize I missed it until well after the time had flowed far past.

As social options open, it will be interesting to see if my perception of time flips back into a more calculated view. Sort of like the difference between how an integral is defined (in math) vs. the way you can calculate the area under a curve using a computer.

Integral vs. its approximation

Something tells me, things will ‘go back to normal’ much faster than I can image — but it may be a new normal. After all, time is one resource that we may be able to optimize, but it will never be abundant.

Creating a temperature tower

When you really want to print some detail in a 3D printed part using plastic filament, you need to understand what temperature the material prints best. Granted, PLA typically likes 190 degrees C and ABS likes about 230 degrees C, but each spool can vary on its optimal print temperature.

That is where a temperature tower comes in. This model has multiple strata of identical elements. Each one is printed at a different temperature. There are several designs out in various repositories, but the one I created looks like:

Temperature tower design

This is shown with an X-ray view so you can see some of the internal components.

This design was based on one I found but I added two horizonal elements on each ‘layer’. One that in 0.5 mm and one that is 1.0mm thick. There is also the temperature value embossed in the layer that it will be printed at – to do this, you need to go in and modify the G-code. Each layer is 5 mm tall. This model demonstrates printing ranging from 220 – 185 (for PLA).

Below are some that I actually printed. One from some white PLA and another in black. You cannot really see the quality differences in the picture, but the white likes to be printed at a slightly higher temperature.

I can tell by the detail loss of the small cones designed into the ends. When it is too hot, they sort of sag. When it is too cold, the layers become more distinct.

I have another temperature tower design for ABS and PETG that covers the range from 260-230.

Once I figure out the right temperature for the material, I write it on the spool.

Now that the printer is working… it is time to give it a real test

I thought I would create a model that really tests it out the capabilities. I designed a country store for this purpose (and placed it out on Thingiverse). The model is HO gage and I will paint it up for the local model train club.

Front of the country store

I figure if the printer can reliably print that fire escape, its working well.

Back of the country store

I printed it, but the fire escape was not as clean as I wished.

The printed result

There were a number of “strings” between horizontal elements that really cause problems. That is probably because I did not have the printer at the optional temperature and retraction enabled.

In order to address this issue, I will need to print a temperature tower, to identify the optimal temperature for this material. That is a whole other post though.

Star wars day — May the 4th be with you

Yes, once again it is Star Wars day and it also happens to be my birthday. Kismet!

If every there was a year to sit down an binge star wars, today is the day. According to the Star Wars Alien quiz (yes, there is such a thing on the link directly above), I am a Wookiee.

Here is a link to a post a few years back on the topic of What if CIOs were Star War’s characters?

A new engraving project – Hufflepuff

Now that the 3D printer is operational again, I needed something else to tackle. So I dusted off the CNC machine — literally.

My daughter is into Harry Potter and has taken the test to determine she is a Hufflepuff. I had sitting around so I thought I would cut out the Hufflepuff crest. Here is what the crest is supposed to look like:

Hufflepuff crest

I only had Poplar available, so ended up creating this:

6″ wooden Hufflepuff crest

Poplar has long fibers and is not all that hard of a wood, so it did not come out as nice as I would have liked. This piece has no real sanding to clean up the edges, but that is easily addressed.

I used a 1mm flat bit to clear away the open areas to a depth of 2mm and a 60 degree V bit for the edges and detail. Overall, I deem the design a success. The next problem is determining how to finish it or to start over with some Maple or other more appropriate wood.