Digital transformation is on the rise across industries, and part of that is creating digital twins of physical assets. A digital twin is a digital representation of a physical object, which can include a 3D model along with other sensor data. A digital twin is used for monitoring, managing, and maintaining a physical asset.
Creating Digital Twins
Drones are playing a critical role in providing data to create digital twins. Getting a camera positioned over every location on an asset provides advantages in speed, frequency, and safety. Drone data is used to create a 3D model using photogrammetry, and inspect the asset.
Capturing quality data with a drone to create a high-resolution 3D model can be difficult, especially for certain assets, such as two massive buildings in an urban environment.
Obtaining professional 3D model outputs requires a pilot to capture images in a specific way, including managing variables such as image overlap, camera angles, sample distance, and image exposure for each building facade. The pilot must balance all these factors while being aware of their surroundings, complying with regulations, maintaining safety, and much more.
So, how do you safely capture the necessary images of a complex site such as two buildings in an urban environment?
The answer is to use flight automation.
Dronelink 3D Mission Plan Preview in Google Earth
Vertical mapping facade missions and crosshatch mapping missions from Dronelink, a flight control software compatible with DJI drones, were used to autonomously capture the data to create a digital twin of the Oklahoma District Court House and Federal Judicial Learning Center and Museum.
Using automated flight was critical in enabling Paper Airplane, a drone solutions company often contracted for inspection, assessment and predictive maintenance projects, to deliver on this historical memorialization and restoration project. The ability to safely and efficiently capture high-quality data enabled the subsequent production of high-quality deliverables that exceeded client expectations.
Paper Airplane pilot flying the mission from the rooftop
Setting the Scene
Paper Airplane, a company specializing in drone services, was hired to create a digital twin of the Oklahoma District Court House and Federal Judicial Learning Center and Museum. The objective of this project was to preserve and restore the historical significance of these buildings.
This site is right across the road from where the Oklahoma City bombing occurred back in 1995. The blast from the bombing damaged 324 buildings within a 16-block radius. The objectives were to inspect the two buildings to assess the structural integrity of damaged areas and plan any preventative maintenance while preserving the historical events damage to the buildings in memoriam.
Instead of relying on binoculars and point-and-shoot ground-based cameras, which offer limited information about the exact location, the building's surface, each image, and related damage, a digital twin 3D model was the ideal solution for all stakeholders involved. The 3D model allows images from multiple perspectives of each point of interest to be overlaid, making it easy to identify and locate any damage.
3D model image with multiple POIs
The goals for the project included:
- Safe operation of drone flights for the capture of all roofs and facades.
- Processing individual images captured into a high-quality 3D representation.
- Export the 3D mesh and point cloud of the buildings.
- Share data with the architectural firm managing the project.
- Help import/implement models for inspection/mitigation and for implementation in BIM and modeling software.
- Provide a cloud-based viewing solution for broad access to personnel who may not have the hardware resources to view the file natively.
- Provide a project file for correlation of the damage to its location on the building surface.
Considerations when Capturing Data for Digital Twins
When creating a plan for producing a high-quality digital twin, we must understand some key factors for capturing the best data possible for use within photogrammetry processing software. Let's first explore what these factors are. Then, in the following sections, we’ll examine why Paper Airplane chose to use the combination of DJI and Dronelink.
Image Quality:
The quality of the imagery captured is of the utmost importance. Key factors of a camera that make a high-quality image are the sensor size, shutter quality, and resolution.
Having a large sensor helps increase the dynamic range of the image. This helps to capture photos in low-lit or shadowed areas with sunlit features within the same frame. These lighting conditions are very common when capturing data around tall structures. It’s also important to have a high-quality mechanical shutter to minimize image distortions or blurs since a drone constantly moves when capturing images. Image stability is also crucial to reduce motion blur and aid camera focus, especially in areas susceptible to high wind currents such as highly urbanized areas.
Image Accuracy:
When creating a 3D model, understanding the image's location can help improve the output mesh's quality and processing times. But in densely urban areas, tall buildings can limit the view of the sky (and satellites) and, in turn, degrade the GPS quality. DJI's Enterprise-level drones use Real Time Kinematics (RTK) to minimize positioning errors in complex environments. The drone's RTK receiver can reference a base station or NTRIP provider for improved positional accuracy. To learn more about RTK and accuracy with DJI Enterprise drones, check out this link.
Asset Coverage:
Every part of the asset needs to be captured from multiple angles for a high-quality reconstruction. This includes capturing NADIR images of the roof and every facade, with enough overlap for processing. Different camera angles help ensure coverage of various features, such as the underside of a balcony. Maintaining proper image overlaps between images horizontally and vertically can be a challenge when flying a facade capture, especially when considering the sample distance, camera angle, and camera sensor size all impact the flight pattern required for maintaining overlaps.
Using flight control software is essential to manage all these data capture parameters for both NADIR mapping flights and facade vertical mapping flights. Automated missions can be created to fly patterns such as vertical columns, stacked horizontal paths, and grid or cross-hatch patterns within a defined boundary using preset overlap values and camera angles.
Spatial Resolution / Sample Distance:
An important factor that impacts reconstruction quality is the resolution. Higher resolution will result in more detail of the object, allowing the 3D model to have more definition of smaller features. The resolution of an image depends on both the camera specifications and the distance from the object being imaged. A term frequently used is Ground Sample Distance (GSD), which is often used in mapping land and, in the case of a building facade, is called Sample Distance. It refers to the dimensions of a single pixel in an image as measured on the asset (for example, 0.10 in/px | 0.24 cm/px).
Resolution and asset coverage are closely interconnected. Determining the desired sample distance and resolution needed will help determine how far away the drone should be from the surface of the asset. This, in turn, determines the required flight pattern needed to maintain front and side overlaps to guarantee asset coverage.
Now that we understand some key factors for creating digital twins let’s talk about the specific hardware and software that can achieve these requirements.
Choosing the Hardware
The DJI Mavic 3 Enterprise is an excellent choice for a scalable hardware solution. With a large 4/3” sensor capturing 20MP images, this drone has an incredible dynamic range to help capture the best images possible. A large sensor partnered with a mechanical shutter that operates at 0.7 seconds puts the M3E best in its class. This faster mechanical shutter allows for faster flight without impacting the image quality, potentially saving time on site. Its small form factor makes it easy to work with, and it has the proper camera system for any digital twin work. Additionally, a long battery life of up to 45 minutes reduces the need for multiple battery swaps during a mission, and limits the total number of batteries needed in the field.
Compared to a consumer drone (Mavic 2 Pro) with a 2-second intervalometer, the M3E performs the same site faster with higher quality results. The Mavic 3 Enterprise can also capture inch-level accurate data while flying safely with a swappable RTK receiver.
Mission Estimates from Dronelink showing M3E capturing the same site 2x faster than the M2P
Automating Data Capture with Flight Control Software
Managing complete asset coverage with high resolution while ensuring safe operations is challenging without using automated flight tools. For this project, Paper Airplane used Dronelink, which offers a suite of flight control tools and workflows to create mission plans and execute them autonomously. Dronelink has the capability to fly facade missions, which are used for vertical mapping, as well as map missions for capturing the roofs.
Dronelink mission preview showcasing the combined mission plans with the facade and map mission types
Within Dronelink, a facade mission can be created easily using a few key inputs. These include the:
- Boundary of the facade (left and right sides, height and minimum safe altitude).
- Vertical and horizontal overlap.
- Target distance from the asset surface and camera sensor being used.
- Camera angle.
Dronelink mission planner and 3D preview showing a single facade mission flying vertical columns and the setup parameters
A facade mission can be created using two different workflows:
- Pre-planning missions in the web or mobile app.
- On-the-fly missions generated in the field using the drone.
Pre-planning facade missions requires creating the mission plan in advance, using the mission planner on a laptop or mobile app. Learn more here. Once a mission is planned, geo-rectification features can align the mission in the field based on a known reference point, such as the corner of a building. Restriction zones can also be added to create no-fly zones. Both features are helpful for improved accuracy and safety.
On-the-fly facade missions allow a pilot to generate the mission in the field, using the drone position to mark the flight boundary area visually, set camera overlaps, and other settings. A pilot is prompted through a series of step-by-step instructions within the mobile app to mark the key inputs that will generate the mission (shown below). Once generated, the mission aligns with the asset and executes one building facade at a time. This feature enables pilots to adjust their plans to account for environmental constraints such as speed, overlap, minimum height restrictions, and other specific areas.
For this site, Paper Airplane chose to run facade on-the-fly missions to capture each facade using two different camera angles and pre-planned map missions for the roofs.
After opening the on-the-fly adaptive facade mission, the pilot flew the drone manually to mark the following key inputs needed, as shown in the image below.
- Step 1: Right corner of the building
- Step 2: Left corner
- Step 3: Top edge
- Step 4: Minimum safe altitude, target distance, and gimbal pitch
- Additional Steps: pattern and overlaps (not shown)
Steps to create a facade mission on-the-fly
After all the required inputs are provided, the mission is generated and can be previewed in 3D before flight. Next, the pilot selects Start, and Dronelink begins to fly the mission.
Automated facade flight showing drone offsets panel
As the mission executes, the pilot can focus on monitoring the drone and its surroundings to ensure safe operations. The drone offsets panel on the right allows the pilot to make real-time inputs during capture, such as minor adjustments to the altitude when getting close to an obstacle. At any point, a pilot can pause the mission and take control. The last position will be saved, whether it is after pausing manually, or for a battery swap, and the pilot can resume the mission from where it left off, maintaining continuity in the data and overlaps for post processing.
Watch a full video walkthrough of the On-the-Fly Facade Mission here.
For this project, Paper Airplane flew 12 on-the-fly adaptive facade missions and two map missions, to ensure complete coverage. To create the required digital twin, each facade face was flown over twice, using a 0 degree, and a -20 degree camera angle. Each facade mission was flown using a vertical flight pattern with 80% vertical and 80% horizontal overlap. A target distance of 40 feet was used to achieve a sample distance of 0.14 in/px | 0.35 cm/px.
A vertical flight pattern was selected instead of a horizontal flight pattern for operational safety reasons.. Due to the surrounding buildings and an urban environment, vertical columns allowed a pilot to stand below the drone as it flew up or down within a column, ensuring that no one else could be directly under it.
Data Processing and Analysis
Once the data is captured onsite, it’s time for post-processing. DJI Terra is an affordable localized processing engine capable of generating high-quality 3D models. The process for generating a 3D model output with DJI Terra can be found here.
When processing data, make sure to include any Ground Control Points (GCPs) that may have been used. GCP’s aren’t required to create 3D models but can help with absolute accuracy and generating tie points. To learn more about GCP’s, click here. During the processing, Manual Tie Points (MTPs) can also be used to build a more dense 3D mesh during the processing.
With the data processed into a 3D model, it is possible for end users (in this case, the general contractor and architects) to review the images and identify the exact position of any damage or areas of interest on the 3D model. This capability is extremely useful to help provide context of where an area requiring attention is located. Measurement tools can determine the exact locations and size of the damaged area and assist in planning any maintenance works.
Final results of 3D model image
Conclusion
Creating digital twins of physical assets using drones and photogrammetry is changing how organizations across all industries are performing work. Making digital twins accessible online to key stakeholders across these organizations saves countless hours, which allows them to be more efficient and helps optimize their own internal workflows.
DJI Enterprise drones combined with flight automation software from Dronelink enabled Paper Airplane to deliver this complex project - and many more - with outstanding results, joining other industry leaders driving the wave of digital transformation across industries.
About Dronelink
Dronelink is a drone flight control software offering automated missions, workflows and flight tools for leading DJI drones. Supported across web, mobile devices, and remote controllers; pilots can fly missions for maps, waypoints, orbits, panos, vertical mapping, or inspections, or use flight modes such as orbit, follow, or focus. Link multiple missions together, automate every detail, preview in 3D, and even unlock enhanced drone precision and accuracy. Learn more at dronelink.com.
About Paper Airplane
Paper Airplane is a cradle-to-grave UAS service provider offering data analytics through survey, inspection, assessment & predictive maintenance. Products are tailored to the client’s individual needs to create deliverables that are unique and powerful yet easy to understand. Paper Airplane offers services in survey, building envelope inspection, utility / ROW inspection, Methane Detection, stormwater inspection / BMP mapping, etc. Learn more at www.pprair.com.
To learn more about additional case studies featuring digital twins, click here.
