Introduction to Camera & Lens Calibration with Agisoft Metashape
Camera and lens calibration is a technique used to understand and correct distortions that occur due to the inherent limitations of camera lenses and sensors as they attempt to map a 3D world onto a 2D plane. It involves estimating the parameters that best describe what happens to a bundle of rays coming from the object when they pass through the lens and onto the image sensor. The following tutorial will show you how to use Agisoft Metashape Pro to solve lens distortions and save the characteristics for photogrammetry and other digital video & photography applications.
Please note, this tutorial assumes some basic knowledge of photogrammetry and general workflow proficiency in Agisoft Metashape.
Video Tutorial
The following is a quick video (no audio) tutorial of the process. For more detailed information, read the subsequent sections for in-depth step-by-step instructions, explanations, and definitions. The sample scan shown is not meant to be a comprehensive scan but merely to show principles outlined in this tutorial.
Lens Calibration in Agisoft Metashape Pro
Please note, this section is summarized and expanded upon based on the Agisoft Helpdesk article, Lens calibration (using chessboard pattern) in Metashape
Metashape can be used for automatic lens calibration that helps to ensure those lens distortions are well simulated by the camera model used in the software. The calibration target acceptable in Metashape is a chessboard pattern on LCD screen, alternatively, a printed chessboard pattern can be used providing that it is flat, and all its cells are squares. The lens calibration procedure supports estimation of the full camera calibration matrix, including non-linear distortion coefficients.
It is recommended to follow this approach for close-range capturing scenarios. It is preferable that the images of the calibration board are taken with the same distance from the camera as the planned distance to the object for 3D reconstruction.
For aerial photography, it is recommended that you calibrate the camera on the reference site using measured ground control points, but the same process outlined here can be used to establish an initial camera/lens calibration.
Capture Photos of the Calibration Pattern
For this tutorial you can use the iPhone14 Pro Max photos provided with the link at the top of the article or if you want to capture your own, follow these steps for capturing photos of the checkerboard calibration pattern:
1. In Metashape menu Tools > Camera > Show Chessboard… command to display the calibration pattern.
2. Scale the calibration pattern so that the number of squares on each side of the screen would exceed 10. Use mouse scroll wheel to zoom in/out the calibration pattern.
3. Use your camera to capture a series of photos of the displayed calibration pattern from slightly different angles. The minimum number of photos for a given focal length is 3.
- Make sure that the focal length is constant throughout the session (in the case of a zoom lens).
- Avoid light glare on the photos. Move the light sources away if required.
- The calibration pattern should take a maximum area of each photo. Move the camera closer to the LCD screen if required.
4. If you are calibrating a zoom lens repeat step 3 for various focal length settings.
5. Right-click on the calibration pattern or press Escape button to return to the program.
6. Upload the captured photos to the computer.
Remove Photo Orientation Tags
By default, most cameras use a small gyroscope to detect the orientation of the camera. In order to properly create a calibration and for photogrammetry in general, we need to remove all orientation tags using exiftool which is a free download.
Download the app and place it in an easy to remember location on your computer. Rename the executable (for windows) to “exiftool.exe” then run the following command from the command line, replacing the path to the executable and the folder containing your images.
"C:\Apps\exiftool.exe" -n -Orientation=0 -overwrite_original "D:\photos\"Embedded Lens Profiles
Some camera and drone manufacturers embed lens profiles into the DNG file which you cannot remove via Adobe software. Other tools such as DxO Photolab or RawTherapee would be needed but Adobe Camera Raw seems to always process raw photography better. Fortunately, we can use ExifTool to remove the camera orientation and lens profile tags at the same time.
Please Note: DJI drones use the OpcodeList3 tag, while Autel drones use OpcodeList3 and OpcodeList2 tags. Check your camera metadata using a tool like ExifPilot for a deep dive into your camera’s metadata. Thanks to Robert Kraczek for the excellent tip.
Here is a script which removes the lens profile from DJI & Autel drone photography & video:
"C:\Apps\exiftool.exe" -n -m -r -progress -Orientation=0 -overwrite_original -IFD0:OpcodeList2= -IFD0:OpcodeList3= "D:\photos\"Load The Photos
To load photos of the calibration pattern:
1. Create a new chunk using Add Chunk toolbar button on the Workspace pane or selecting Add Chunk command from the Workspace context menu (available by right-clicking on the root element on the Workspace pane).
2. Select Add Photos… command from the Workflow menu.
3. Browse to the folder containing the photos of the calibration pattern and select files to be used. Then click Open button.
4. Loaded photos will appear in the Photos pane.
Calibrate the Camera Lens
To calibrate the camera lens:
1. Open Camera Calibration… dialog available from the Tools menu and make sure that Camera Type setting corresponds the lens to be calibrated. Use Fisheye value for fisheye lens and Frame camera for prime lens or zoom lens.
2. Select Tools > Camera > Calibrate Lens… command.
3. In the Calibrate Lens dialog box, select the desired calibration parameters. Click OK when done.
4. The progress dialog box will appear displaying the current processing status. To cancel processing, click the Cancel button.
5. The calibration results will appear on the Adjusted tab of the Camera Calibration… dialog available from the Tools menu.
Once the calibration procedure is completed it will be possible for you to observe the detected chessboard corners on each photo (the photo can be opened by double-clicking on its name in the Photos pane). Most of the corners should be detected correctly, if not — we recommend collecting a new set of calibration images and re-run the procedure.
For each detected corner, the reprojection error is displayed: it characterizes the difference between the detected corner position and its’ position estimated according to the calculated calibration. The errors are scaled x20 times for display.
The detected corners are marked with red, blue, and gray points. Red and blue points are used for calibration, while gray points are not. It is important that each color point has points with a different color as its nearest neighbors.
Use Save button on the Adjusted tab to save the adjusted values for camera calibration parameters to file. The saved lens calibration data can later be used in another chunk or project for the actual dataset captured with the same camera and lens.
Using Fixed Camera Calibration Parameters
Please note, this section assumes your scanned subject is focused to infinity or your lens is subject to a minimal amount of focus breathing and shot at a similar range/distance to the subject. If your subject is shot at a different distance or your lens is susceptible to focus breathing, follow this step, run alignment then proceed to the next sections on What is Focus Breathing? and Using Variable Camera Calibration Parameters
After you have saved the calibration parameters for the lens, you may proceed with the workflow steps in a separate chunk (or in a separate project) for an actual dataset captured with the same camera and lens. If your subject is shot at the same distance as your calibration with a good lens, this method can yield superior results.
1. Create a new chunk in the current project (where the camera calibration procedure had been performed) or open/create a project.
2. Open the dataset using Add Photos command on the Workflow menu for the new chunk.
3. Load calculated camera calibration parameters. Open the Camera Calibration dialog from the Tools menu and use Load button on the Initial tab.
4. To protect the calibration data from being refined during Align Photos process on the Initial tab use Select… button for the option Fixed parameters: in the Fixed parameters dialog window choose Check all box (alternatively you may select only those parameters that you want to be preserved during alignment procedure). Thus, initial calibration values will not be changed during the alignment processing step.
5. Press the OK button to confirm and proceed with the general workflow starting with the Align Photos step.
What is Focus Breathing?
Please note, this section is summarized and contextualized for photogrammetry from the excellent article, Understanding Focus Breathing by Nasim Mansurov
Focus breathing is a term used in photography to describe the change in focal length that occurs when you adjust the focusing distance of a lens. This phenomenon is common in many modern lenses. As you adjust the focus from a close-up to infinity, focus breathing can cause noticeable changes in both the angle of view and magnification. These changes can often be clearly seen by the photographer during the focusing process. Different lenses can exhibit varying levels of these focal length changes. It’s important to note that focus breathing can occur on both prime and zoom lenses, will always result in a decrease in focal length and never when focused to infinity.
Why is this an issue? Changes in focal values can affect alignment of camera views in photogrammetry thus affecting the final output. It’s also especially important for film and television production as focus breathing is distracting and undesirable.
Here is a YouTube video primer on lens/focus breathing.
Modern lenses often have an internal focus design. This means that when you adjust the focus, only some parts inside the lens move, not the whole lens. This is good because it makes it easy to use filters, the lens doesn’t get physically bigger or smaller when you focus, and focusing is faster and quieter. Unfortunately, these lenses can exhibit focus breathing when the focus changes the lens’s optical properties, including its focal length. The amount of change can be small or large, depending on the lens design.
The simplest way to determine if a lens has focus breathing issues is to examine its specifications, particularly its maximum magnification. In simple terms, maximum magnification is the ratio of the image size on the image sensor to the actual size of the subject. This data can be readily used to ascertain how much focus breathing a lens displays when compared to other similar lenses. The smaller the maximum magnification, the more severe the focus breathing effect. In general, if your lens has a maximum magnification of greater than 0.20x this will minimize the effect.
This effect can be mitigated in photogrammetry in two ways. Frame the subject wide, with a low aperture and focus set to infinity or shoot in “focus groups” by using the autofocus once on the subject then set the focus to manual then maintain your distance to the subject while remaining within the focus range. When you need to change your distance to the subject, be sure to grab a quick shot of your hand so later you know when focus groups have changed. The second option can be a little tricky in Metashape as you need to create multiple focus calibration groups for every scanning session then calibrate the focus groups separately. This technique can make both capture and processing much more tedious than it needs to be.
Some newer cameras such as the Sony A7rIV and A7rV have focus breathing compensation features. When enabled, it minimizes the field-of-view shift caused by focus breathing. However, it achieves this by slightly cropping the image — approximately 80% of the original area or a 1.25x linear crop. The image remains consistent in size, but the expansion or contraction is reduced. While it can minimize the effect, it’s cropping the image which is not desirable in photogrammetry.
Calibrating a lens with an emphasis on correcting focal breathing is an advanced and challenging process. In general, the easiest way to minimize the effect is to choose your lenses wisely by considering the lens specification many people ignore.
Using Variable Camera Calibration Parameters
The iPhone 14 Pro cameras are susceptible to a lot of focus breathing. After following the steps in Using Fixed Camera Calibration Parameters, you’ll see that when you run alignment, there’s a severe distortion in the output. You’re asking yourself, “what gives? I calibrated the lens, and the result is wrong. Metashape’s lens calibration doesn’t work”.
If we check the initial focal length values (f) measured in pixels, of the checkerboard images and compare them with the scan images, there is a huge disparity between the two. This means there could be considerable focus breathing despite the same camera and a fixed lens being used.
Remember, checkerboard calibration only works well in this case if the focus is set to infinity, or your lens has minimal focus breathing and you’re shooting from the same distance. As these two scan sessions are so different, it yields the fixed calibration useless. It doesn’t mean the calibration is wrong, only that the scan should be under the same conditions as the checkerboard calibration. While this can work in controlled studio conditions, unfortunately, when in the field this can cause issues and it’s not practical nor feasible to always know what distance and focus value corresponds to your fixed calibrations. So, what do you do?
Firstly, it is best to shoot in groups at the same distance as described above, focus to infinity, or minimize focus changes, and maintain distance to the subject. This will create natural breaks you can group your calibrations by. Secondly, we can allow Metashape to vary the focal length values while maintaining other camera lens intrinsics as long as the focus breathing effect is not dramatic, or our focal values are within a limited range. This is not ideal, but it can allow for some variation and generally returns fewer errors in alignment.
Click on Image-Variant parameters, then select the f checkbox. Close the dialogs, then go to Tools > Optimize Cameras and select “Adaptive camera model fitting”.
Adaptive camera model fitting is an option in Metashape that enables automatic selection of camera parameters to be included into adjustment based on their reliability estimates. For data sets with strong geometry, it helps to adjust more parameters during initial camera alignment. For data sets with weak camera geometry, it helps to prevent divergence of some parameters1. If you enable adaptive camera model fitting, Metashape will try to define automatically which parameters of the interior orientation should be adjusted. If you disable this option, F, Cx, Cy, K1, K2, K3, P1 and P2 will be adjusted during the alignment.
When we open the camera calibration window, you will see each image now has individual focal (f) values. If you see values far outside a range by more than a few hundred pixels, disable/reset that camera and run Camera Optimization again or allow all the images to find lens unique distortion coefficients.
Select all the additional image variant parameters from above and run Camera Optimization again with adaptive camera model fitting enabled. Beware, this will allow every image to find new values and can affect your overall alignment but if you have a lot of images with severe focus breathing, your alignment should improve.
Additionally, using gradual selection and filtering tie points by reconstruction error and uncertainty will improve your alignment and improve your camera/lens solve.
It is recommended to review the adjusted values for the cx, cy, b1 and b2 parameters in the Camera Calibration dialog. If the values for the parameters are too large (hundreds or more for cx, cy and tens or more for b1, b2), then it may be reasonable to re-align the data set having these parameters fixed.
Usually, the values for cx and cy are around a couple of dozens of pixels (or less), as the values are measured from the center of the image. And the values for b1 and b2 are usually not more than a few units.
If the values are hundreds or even thousands — it indicates that the estimation of the calibration parameters is probably incorrect. Sometimes it may be related to the flight mission specifics, for example, if the camera is not rotated by 180 degrees on the neighboring flight lines. You can try to fix the mentioned parameters and re-align the data set having “adaptive camera model fitting” option disabled.
The most obvious way to solve this is to create calibration groups by the camera & lens combination and group the images by focal values (f) with similar ranges which are contained in the metadata. Unfortunately, this requires some work in python and beyond the scope of this tutorial. Stay tuned for a future tutorial which will cover this subject.
The important thing to take from this tutorial is to consider your choice of camera and lens combination more carefully, how focus breathing works and if you’re getting bad alignments with strong shooting techniques and overlap, there may be nothing wrong with the software or camera, you just need to consider more things you might not have in the past. It is possible to get professional results, even with an older camera or iPhone, if you understand these principles and know how to deal with them.
This concludes the tutorial on Introduction to Camera & Lens Calibration with Agisoft Metashape.