Over the past 20 years, more than 7,300 natural disasters were recorded worldwide, affecting more than 4 billion people across the globe and generating $2.97 trillion in economic losses (UNDRR).
Manual Entry Wasn’t Made For 21st Century Crises
Over the past 20 years, more than 7,300 natural disasters were recorded worldwide, affecting more than 4 billion people across the globe and generating $2.97 trillion in economic losses (UNDRR). The colossal effort to manually collect, track, translate, store, verify, and share critical data in the wake of these disasters has created overwhelming, and at times, insurmountable challenges for humanitarian responders— jeopardizing accurate decision making during a crisis and hindering global ability to prevent and abate future disasters. That’s where DEEP was born.
Remove Collaboration Barriers, Improve Response Time, Save More Lives
DEEP is a platform built with AI in mind to centralize, accelerate, and strengthen inter-agency response to humanitarian crises at national and field levels. The free, open source tool was developed by field responders in the wake of the devastating 2015 Nepal earthquakes, and has since become a go-to resource for leading global humanitarian organizations, including UNHCR, UNICEF, UN OCHA, and the IFRC. Today, DEEP hosts the largest analysis framework repository in the international humanitarian sector, hosting more than 85,000 carefully annotated response documents and connecting more than 3,000 expert users worldwide.
Since its inception, DEEP has been used to inform more than 1,800 international humanitarian projects whose scopes are estimated to impact more than 98 million people, including USAID’s response to COVID-19, ACAPS’s response to the Rohingya crisis, and UNHCR and IFRC’s response to the Venezuela migrations crisis. More than two-thirds of individuals in areas where DEEP is utilized earn less than $5 USD per day.
Partners
The COVID-19 pandemic has presented unprecedented challenges to the ability of humanitarian actors to collect and analyze data in humanitarian responses around the world. In addition to worldwide stay at home orders that have constrained traditional responses, humanitarian organizations have faced five concrete challenges:
To date, DEEP is active in coordinating COVID-19 responses in 14 countries across Central and Eastern Africa, the Middle East, Southeast Asia and Latin America. In the case of iMMAP, DEEP is centralizing COVID-19 response data collection among 464 humanitarian organizations spread across 6 target countries and operating alongside broader global efforts.
Through this partnership, DEEP has directly strengthened assessment and analysis capacities that continue to inform response and resource allocation for more than 41.6 million people in targeted areas.
More than 50 million people around the world are internally displaced. Forced by natural disasters and violence to flee their homes and often living in refugee-like conditions within their own country’s borders, the UNHCR has called internally displaced people “among the most vulnerable in the world.”
Each year, thousands of internal displacement crises are tracked and analyzed by the Internal Displacement Monitoring Centre (IDMC)—the world’s most definitive source of data on internal displacement. Prior to partnering with Data Friendly Space (DFS), IDMC relied on a small IT team to collect and securely store reliable, timely, and longitudinal data on millions of displaced people from over 188 countries. This limited capacity held back the international community’s ability to respond to, report on, and ultimately, reduce the risk and impact of internal displacements across the globe.
Since 2019, DFS has introduced multiple data management tools backed by AI to better process, store, and analyze internal displacement trends and enable faster, more robust humanitarian responses in the field. Highlights include:
DFS Solution | Result |
Efficient Media Monitoring | DFS helped to replace IDMC’s manual processing with IDETECT, a tool that analyzes thousands of global news sources daily through natural language processing. With IDETECT, IDMC now automates data collection on natural disaster and armed conflict displacements and can visually present the data to partners, funders, and decision makers worldwide. |
Smart Reporting | DFS carried out a complete re-development of the main IDMC web application used to collect, analyze, and report on internal displacement data around the world. HELIX, a global information management system, is now pre-populated from the IDETECT natural language processing system, resulting in: 1. Reduced manual input and increased capacity for team members 2. More accurate and comprehensive data entry DFS’s data outputs are used to create reports like IDMC’s 2020 Global Report on Internal Displacement that are a primary resource to the United Nations and humanitarian leaders around the world. |
21st Century Security | DFS redesigned and deployed an entirely new infrastructure to securely store IDMC’s data from 188 countries and thousands of annual displacement events. DFS continues to provide around the clock security monitoring for IDMC’s online resources. |
Seamless Integration | DFS managed the migration of data from thousands of past internal displacement events to a centralized reporting system. Upgrades are designed to increase the organization’s interoperability and dramatically reduce human error and the need for manual input. |
DFS’s data management tools, including Smart Reporting (Helix) and Efficient Media Monitoring (IDETECT), were utilized to produce multiple reports on the state of internal displacement during the COVID-19 pandemic, including Internal Displacement 2020: Mid-year update. The report analyzed 14.6 million new internal displacements across 127 countries in the first 6 months of 2020, giving the humanitarian community access to geolocalized data on the key priorities of people in need for a more accurate response on the ground.
DFS also built the data processing tools to enable detailed tracking and analysis of key variables impacting internally displaced people throughout COVID-19, including the virus’ impact to health, livelihoods, housing, education, and security. Learn more about IDMC’s response to COVID-19.
Companion note to an Excel demonstration workbook
Aldo Benini
Jose Cobos Romero
27 January 2022
Authors:
Aldo Benini
José Cobos Romero
27 January 2022
Summary
Possibility Theory, a variant of probability theory, is well suited to analyze large-N measurements of variable reliability, obsolescence and redundancy. Very few applications are known from the humanitarian sector, and these are about stocks and flows of relief goods measured on continuous scales.
We demonstrate the usefulness of Possibility Theory in testing the binary hypothesis that the severity of humanitarian conditions in a given region and sector is high / not high. The test relies on numerous ordinal severity ratings produced by coders who review humanitarian reports with location, time, sector, affected group and context information. The statistic of interest is a measure of confidence that the true severity is high / is not high. This measure is continuous bounded in the interval [-1, +1] and thus easy to visualize in tables and maps. Also, we propose an allied measure of information gaps.
The demonstration data are from Colombia during a 14-month period in 2020-21 (18 May 2020 – 30 June 2021). Two organizations, iMMAP and Data Friendly Space (DFS), collected, excerpted and processed relevant documents of various types in a dedicated database application known as DEEP. In 2021, DEEP projects were operational in thirty countries. In Colombia, it delivered the information base for the Humanitarian Needs Overview (HNO) 2021.
The iMMAP / DFS coders parsed 357 documents (“leads”) and turned them into 1,540 DEEP records (“entries”). From these we derived 24,920 observations each with a location, publication date, sector and rating on a 5-level severity scale. Our algorithm produces confidence and information gap estimates at the aggregate level (pairs of Department [Admin1] and sector). All 33 Departments and 11 sectors are represented, although not for all pairs.
The Excel demonstration workbook uses the data from a subset of six Departments, keeping 10,624 observations. The workbook architecture is such that the user can change parameter settings, view the updated outcomes in Department X sector tables, and compare them to the ones under the initial settings, all in the same sheet. Those interested in the inner workings find explanations in column header comments and in the “back office” sheets that do the work of aggregation and gap measure calculation.
This companion note details the motivation to turn to Possibility Theory, minimal generic elements of the theory, our choices in adjusting for reliability, obsolescence and redundancy, the steps leading to the confidence measure, as well as the components of the gap measure. The note then describes the data generation and analysis workflow in the DEEP institutional environment. It presents select results from the whole-Columbia dataset. It explains the workbook structure and the functions of the various parameters, as well as some less common types of formulas. Deliberately, no VBA programming is involved.
Together, note and workbook demonstrate how an infrastructure like DEEP and unusual tools like Possibility Theory open a way to combine large sets of structured humanitarian observations in a quantitative severity measure. Our measure captures the degree of uncertainty in a more informative way than the usual rank order-based statistics for ordinals. In addition, the gap measure points to regions and sectors needing further investigation or needs assessments.