Creative Commons License
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.
Abstract
With funding from the Sloan Foundation and Harvard Dataverse, the Harvard Center for Geographic Analysis (CGA) has developed a big spatio-temporal data visualization platform called the Billion Object Platform or "BOP". The goal of the project is to lower barriers for scholars who wish to access large, streaming, spatio-temporal datasets. Since once archived, streaming data gets big fast, and since most GIS systems don't support interactive visualization of millions of objects, a new platform was needed. The BOP is loaded with the latest billion geo-tweets and is fed a real-time stream of about 1 million tweets per day. The CGA has been harvesting and archiving geo-tweets since 2012. As tweets flow into the BOP, they are enriched with sentiment and census information to support further analysis. Incoming and intermediate data is streamed/stored in Apache Kafka. The core of the BOP is Apache Solr, which supports fast search. Some significant enhancements were done to Solr (and contributed back) -- notably 2D "heatmap faceting" to support spatial visualization. The BOP fronts Solr with a RESTful web service, which provides a friendly, and secure API that is accessed from a browser-based client. The client developed, dynamically displays temporal and spatial distributions of results for result sets containing hundreds of millions of features. The system is open source and runs on commodity hardware. It is hosted on Massachusetts Open Cloud (MOC), an OpenStack environment. All components are deployed in Docker orchestrated by Kontena.
DOI
https://doi.org/10.7275/R5ST7N0G
Recommended Citation
Kakkar, Devika; Lewis, Ben; Smiley, David; and Nunez, Ariel
(2017)
"The Billion Object Platform (BOP): a system to lower barriers to support big, streaming, spatio-temporal data sources,"
Free and Open Source Software for Geospatial (FOSS4G) Conference Proceedings: Vol. 17, Article 15.
DOI: https://doi.org/10.7275/R5ST7N0G
Available at:
https://scholarworks.umass.edu/foss4g/vol17/iss1/15
Included in
Communication Technology and New Media Commons, Databases and Information Systems Commons, Geographic Information Sciences Commons, Graphics and Human Computer Interfaces Commons, Human Geography Commons, Nature and Society Relations Commons, Numerical Analysis and Scientific Computing Commons, Other Geography Commons, Spatial Science Commons, Urban, Community and Regional Planning Commons