BY MATTEO LUCCIO / CONTRIBUTOR / PALE BLUE DOT LLC PORTLAND, ORE. / WWW.PALEBLUEDOTLLC.COM
The waves created by Google’s announcement that it will stop supporting Google Earth Enterprise (GEE) in the spring of 2017 continue to ripple through the geospatial community. For this third installment in this series, I interviewed Andrew Hill, Chief Science Officer at CartoDB, a software vendor; John-Isaac Clark, Director, Product Management – Geospatial Big Data at DigitalGlobe, a satellite imagery and information vendor; and Lt. Col. Michael Russell, GEOINT Visualization Services (GVS) Program Manager at the National Geospatial-Intelligence Agency (NGA), one of the largest users of GEE.
FIGURE 1. A high off-nadir image of the San Francisco area, taken on February 7, 2016, courtesy of Digital Globe.
CartoDB: Like GEE, but without 3D
3D GLOBE Several geospatial companies offer partial replacements to GEE, though their offerings differ somewhat from each other and from GEE. The biggest difference between CartoDB and GEE is that the former lacks a 3D globe. Instead, like Google Maps, it is focused on flat projections. CartoDB and GEE, however, share many features, Hill points out, including the ability to create multiple maps from a single dataset and for teams to collaborate around geospatial data. “GEE’s portability is a very valuable feature for enterprise users,” says Hill. “Many GIS tools are pretty heavy. They have many requirements about where they can be installed. Many of them don’t work very well on mobile devices. CartoDB solves many of the same problems [as GEE].” CartoDB is a software-as-a-service, so it enables users to build apps and analyze data directly from their browsers, without installing any tools, and to access their work from mobile devices.
FIGURE 2.
FIGURE 3.
FIGURE 4.
FIGURES 2-4.
Deep Insights demo images show how a company can identify the locations of its target demographics using the locations of its retail shops and census tract information. This shows a target market in New York City.
Both GEE and Esri’s ArcGIS Earth are built around the idea of presenting geospatial data on a 3D Earth and allowing users to navigate on its surface. By contrast, Hill says, “CartoDB is focused around drilling down into geospatial data and then gathering insights from that, but then also sharing those insights with other people.” He points out two important differentiators between CartoDB and the other two platforms. First, that the data in CartoDB resides on “a powerful back end” that can analyze that data and perform filtering, measurements, and other GIS functions. “Combining those features is what allows enterprise users to tap into and gain insights from their data,” says Hill. Additionally, he says, CartoDB’s ease of use allows people with very different backgrounds, including many without training in GIS, to look at their data, find patterns and outliers, and make some predictions.
CartoDB, Hill explains, has two major user groups, each defined by one subset of CartoDB’s functionalities. The first group consists of university users and people who use the free version of the software. “They tend to use CartoDB as a mapping platform, so they will load data into it, work with advanced cartography, and then publish a map,” says Hill.
The second group consists of enterprise users, who do such things as build analysis dashboards, integrate CartoDB’s geospatial back end with their own tools, and leverage the CartoDB platform at scale in their teams. “We have people who use CartoDB for everything from building geospatially-enabled data portals to building store locators, to performing analytical analyses with demographic data.”
Like GEE, CartoDB enables users to visualize their geolocated data on a map. “It’s pretty incredible how powerful that can be for enterprise users as the first step,” says Hill. “Many users have data that contains geospatial information that they have only ever seen in a spreadsheet.” Also like GEE, CartoDB enables users to run analyses on datasets that are changing in real time. “You could be looking at such things as clustering of customers, or the overlap of sales regions, or the performance of different neighborhoods in terms of real estate values and publishing such valuable maps internally,” Hill explains.
NGA’S TAKE ON GOOGLE’S DECISION
NGA has been utilizing GEE mostly on a classified level. “This is one of the major reasons why Google decided to distance itself from the U.S. Intelligence Community (IC) and from supporting the enterprise server the way it did,” says Russell. “Google wanted to change its code framework and appears to have determined that the separate instances they would be required to support in a system set up specifically for the Department of Defense and the IC was not in its best interests from a corporate perspective.”
If NGA was only going to utilize Google Earth in an unclassified way, he explains, the agency would simply stick to the free, public version of the platform and continue to run its programs on it. However, only a small percentage of U.S. military and intelligence users around the planet—generally those focused on humanitarian or emergency and disaster relief efforts—use Google Earth on the unclassified level. Most of them utilize the version of Google Earth that NGA makes available to them on the Department of Defense’s secret network, SIPRNet; many of them also use it on the top secret JWICS network.
FIGURE 5. Crime in London shown with Map Layer Wizard. CartoDB Editor allows users to select a basemap, select the type of map to create, and customize the style and information displayed in the infowindows.
FIGURE 6. Worldwide reaction on Twitter to Google’s announcement of Alphabet. The CartoDB Editor allows users to style, annotate, and animate maps via the Editor UI or the CSS
The Google Earth version that NGA supports and that the GVS program office provides to the user community is accessed by 30,000-35,000 individual users every month and generally gets close to one billion hits per month by those users, according to Russell, sup- porting as many missions as there are unique users. He defines a “hit” as “any time a user changes an overlay or any particular set of imagery tiles during their workflow.”
On all security domains, GVS offers what it calls a “GIS portal,” which serves as a collaboration environment. However, Russell points out, it is only a 2D environment. “From a 3D perspective, really, the NGA GIS has been built on a Google Earth foundation.”
NGA’S TRANSITION OFF OF GEE
NGA has a longstanding collaboration with ESRI in the GIS field. GVS offers the IC Esri’s 3D-capable viewer ArcGIS Explorer. “It is more of an analytical tool, not widely used like Google Earth,” says Russell. While Google Earth provides a “common operating picture” for operation centers around the world, he explains, ArcGIS Explorer has been used mostly by analysts for analytical and exploitation activities and is more commonly focused on a particular area. As its main replacement for GEE, NGA will adopt Esri’s new platform, ArcGIS Earth. The way GVS will be using it, Russell says, is as “sort of a follow-on to ArcGIS Explorer.” It is currently in pre-beta testing with some of NGA’s major users around the planet and will behave very similarly to Google Earth once it is fielded, he says. (Editor’s Note: This series featured ArcGIS Earth in our Fall 2015 issue.)
Many of NGA’s users are very heavily dependent on Google Earth because over the past decade, they have built many services and data overlays on it using very loosely specified KML that is not supported by other 3D visualization providers. “So, while they display well on Google, they do not display well on other 3D applications,” says Russell. Handling that non-specific KML has been a focus area for the GVS development team, with help from ESRI personnel, so as to provide access to it for users over the next couple of years, during the transition from GEE to ArcGIS Earth. “It is going to take that long because a lot of development time and transition has to be built in for handling some of our larger customer needs,” Russell explains. To support this transition, GVS has teamed up with Thermopylae Science & Technology (the company where Clark worked for nearly nine years prior to moving to DigitalGlobe), to keep GEE “on life support.” This will allow users to continue to access it and the applications they built on it, while transitioning to ArcGIS Earth, Map of the World 3D on Cesium, or another 3D environment. “ArcGIS Earth will have the closest look, feel, and behavior to what Google Earth provides today,” says Russell.
In preparation for moving off of GEE, GVS is pursuing two different applications: a desktop installation, similar to the current public version of Google Earth, and a Web-based client. The former is aimed at users in disconnected environments, who need to be able to run a small portable globe that GVS develops and sends them. “We send portable globes all around the world with .5TB to 2TB of data (and sometimes more) to many of our disconnected users who would otherwise have no access to a geospatial environment for their operational and tactical planning needs,” says Russell. That, he adds, is another reason why GVS offers ArcGIS Earth as an install.
The Web-based client it is developing is based on Cesium and will provide access to NGA’s Map of the World environment, which integrates all GEOINT- related and multi-source content through twelve different data views. This option will not require users to install a client on their device, but does require a network connection. “They are both being built to have an advanced KML exposure capability in order to transition users smoothly off of Google Earth and into another realm,” Russell says. “The Defense Information Systems Agency wants to ensure that future users stop developing KML and start developing more OCG-stringent data services. However, based on the number of KML services that have been developed and their scope and size, it would be unfair to have our user community re-write or re-accomplish all of those services they have already built into some new format.”
While GVS’ primary focus in replacing GEE was to make sure that it met all of the requirements of its current users, the transition does offer NGA the opportunity to extend some capabilities. “Many of the advanced capabilities and exploitation capabilities are being developed by the exploit program office and a soon-to-be-awarded analytic service contract,” Russell says. “Additionally, we are partnering with the National Reconnaissance Office to leverage some of what they have been able to develop in their open source version of Cesium, the Multi-Int Spatial and Temporal (MIST) Toolkit in 2D and 3D, to increase the capabilities we have available in Map of the World. A lot of what we are looking to leverage is more of the heavy analytical functions we don’t currently have within our visualization tools.” Early on in the GEE replacement effort, GVS also addressed the need for 4D visualization—that is, 3D plus the temporal perspective—to enable users to look at how objects or targets change or behave in relation to each other over time. With aircraft and satellites flying over specific targets or covering large swaths of the Earth, this time slider technology enables them to see where things were and what was happening on the globe at different points in space and time. “Temporal analysis has been in increasing demand from many of our user groups over the past year,” says Russell.
FIGURE 7. NGA Director Robert Cardillo speaks at GEOINT Symposium, 2015.
3D SPATIAL ANALYSIS AT PLANETARY SCALE
3D view has been a standard in industry for more than a decade now. Recent development efforts by Esri, VRICON, and others have demonstrated “an exciting amount of growth in the number of people who recognize the need for 3D,” says Clark, who had predicted this trend in the spring of 2015. These capabilities are coming to market as the deadline for deprecation of the GEE approaches. “I think that it is exciting to see,” he says. “It has raised awareness about 3D and its role in spatial.”
Recently, Clark points out, there has been an upsurge in interest and capabilities with regards to remote sensing and spatial analysis at planetary scale, including DigitalGlobe’s development of its Geospatial Big Data Analytics Platform (GBDX), the work of Rebecca Moore’s Earth Engine team at Google, and the rise of microsatellite companies, such as Planet Labs. They are “gaining answers and insights, not just pixels,” he says. “It is going to be interesting to see how that type of information fits into some of these new 3D tools. What does a 3D picture look like when we can start doing such things as monitoring and change detection over days, weeks, and months? Some fascinating things are coming with processing information at scale and then putting it in a 3D environment.”
DigitalGlobe is making all of the 90+ petabytes of data in its archives accessible in the cloud via its GBDX platform, enabling anyone to extract knowledge from them. “I wonder how capabilities like ours will play in 3D GEE replacement capabilities,” says Clark. “Once there are a couple of 3D products out there, I think that people are going to want to do things with 3D that they were never able to even contemplate doing before.”
IS SPATIAL SPECIAL?
While geospatial is increasingly just another dimension of data, the real challenge has been to use it correctly, according to Hill. As an example, he cites the way we measure populations. “They are statistical estimates from the aggregation of people in administrative units, which don’t really adhere to predictable shapes and sizes in geographic space. They have different sizes, different relationships with their neighbors, and all these characteristics that make them challenging to analyze in reliable ways.” Previously, he points out, only GIS experts knew methods to deal with this challenge, but now data scientists increasingly have access to those tools, and GIS in business enterprises is getting closer to being a data science. “At the enterprise level, we are going to see more and more GIS practitioners doing more broad data science. And geospatial will be incorporated in everybody’s data science.”
DATA SCIENTISTS INCREASINGLY HAVE ACCESS TO THESE TOOLS, AND GIS IN BUSINESS ENTERPRISES IS GETTING CLOSER TO BEING A DATA SCIENCE.
– Andrew Hill, CartoDB
So far in this series we have featured ArcGIS Earth, VRICON, INdicio (by Galdos Systems Inc.), Skyline Globe and Terra Explorer (by Skyline Software Systems), Onix Networking Corp. (which integrates Skyline software into organizations’ IT systems), CartoDB, DigitalGlobe, and the NGA. In future articles, we will explore further the offerings of DigitalGlobe, including their newly launched Maps API, and continue to explore the constantly changing landscape of geospatial vendors, products, and large users and the percolating repercussions of Google’s decision to end support for GEE.
