Big Data Strategies for Government, Society and Policy-Making

Abstract

The paper aims to facilitate a discussion around how big data technologies and data from citizens can be used to help public administration, society, and policy-making to improve community's lives. This paper discusses opportunities and challenges of big data strategies for government, society, and policy-making. It employs the presentation of numerous practical examples from different parts of the world, where public-service delivery has seen transformation and where initiatives have been taken forward that have revolutionized the way governments at different levels engage with the citizens, and how governments and civil society have adopted evidence-driven policy-making through innovative and efficient use of big data analytics. The examples include the governments of the United States, China, the United Kingdom, and India, and different levels of government agencies in the public services of fraud detection, financial market analysis, healthcare and public health, government oversight, education, crime fighting, environmental protection, energy exploration, agriculture, weather forecasting, and ecosystem management. The examples also include smart cities in Korea, China, Japan, India, Canada, Singapore, the United Kingdom, and the European Union. This paper makes some recommendations about how big data strategies transform the government and public services to become more citizen-centric, responsive, accountable and transparent.

1. Introduction

In 2018, over 2.5 quintillion bytes of data were created every day, and by 2020, it is estimated that 1.7 megabytes of data will be created every second for every person on earth (DOMO, 2018). With each click, share, like, and swipe, society is creating big data. Every day, billions of people interacting with their devices via the internet contribute to a world of valuable information.

Advances in technologies and the increasing amount of information are transforming how business is conducted in many industries, including government. Government data generation and digital archiving rates are on the rise due to the rapid growth of mobile devices and applications, smart sensors and devices, cloud computing solutions, and citizen facing portals. As digital information expands and becomes more complex, information management, processing, storage, security, and disposition become more complex as well. New capture, search, discovery, and analysis tools are helping organizations gain insights from their unstructured data. The government market is at a tipping point, realizing that information is a strategic asset, and government needs to protect, leverage, and analyze both structured and unstructured information to better serve and meet mission requirements. As government leaders strive to evolve data driven organizations to successfully accomplish mission, they are laying the groundwork to correlate dependencies across events, people, processes, and information.

The use of laptops, smartphones, and tablets in government continues to grow. Mobile computing enables effective telework and supports continuity of operations and worker productivity in disaster recovery. The communication technologies of social media business technologies are enabling citizens to take a proactive role in government. As social media skills are increasingly well honed, the value of this collaborative technology is maturing, especially when used to enable open, transparent government and to facilitate government service delivery. In the long term, these technologies will be key tools for dealing with the complexity of increased digital information. Big data is one of the intelligent industry solutions and allows government to make better decisions by taking action based on patterns revealed by analyzing large volumes of data-related and unrelated, structured and unstructured.

In recent years, the amount of data generated worldwide is increasing at an astounding pace, by 40% annually (GetSmarter, 2018). Big data analytics can be an immensely powerful tool to monitor how services are performed, increasing transparency and allowing for better results, processes and decisions. However, many public services are not able to fully manage and analyze the information to benefit from its immense potential. In this regard, by identifying the opportunities and challenges that governments face in the context of big data and open knowledge, this paper intends to create synergies and cooperation between governments and public-service agencies, leading to more effective and informed actions by public administrations.

Accordingly, this paper aims to: 1) identify concrete big data and open knowledge opportunities and requirements in public administrations and in specific policy contexts, and 2) promote cooperation among governments and public service agencies in order to accelerate the data-driven transformation.

2. Opportunities and Challenges of Big Data

2.1. Definitions of Big Data

According to International Data Corporation (IDC), IDC defines big data technologies as a new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data by enabling high velocity capture, discovery, and analysis. New capture, search, discovery, and analysis tools can help organizations gain insights from their unstructured data, which accounts for more than 90% of the digital universe (IDC Government Insights, 2012). The definition encompasses hardware, services, and software that integrate, organize, manage, analyze, and present data that is characterized by volume, variety, velocity, and value extracted and leveraged from the data to further mission.

Big data was mainly used as a term to refer to the size and complexity of data sets, as well as to the different forms of processing, analyzing and so forth that were needed to deal with those larger and more complex data sets and unlock their value (I-Scoop, 2020). Most people used to look at the pure volume and variety perspective: more data, more types of data, more sources of data and more diverse forms of data. But data as such is meaningless, as is volume. What really matters is meaning, actionable data, actionable information, and actionable intelligence. A goal of big data strategies is the action to get there and move from data to decisions and actions, thanks to big data analytics and artificial intelligence.

Amid all these evolutions, the definition of the term big data has been evolving, moving away from its original definition in the sense of controlling data volume, velocity and variety (META Group Inc., 2001). A key question in that predominantly unstructured data chaos is what the right data that we need to achieve one or more of possible actions are. The creation of value from big data and of data and information overall is a holistic one, driven by desired outcomes. Big data is high volume, velocity and variety information assets that demand cost effective, innovative forms of information processing for enhanced insight and decision making (META Group Inc., 2001). With the Internet of Things (IoT) happening and the ongoing digitization in many areas of society, science and business, the collection, processing and analysis of data sets and the right data is a challenge and opportunity for many years to come.

The renewed attention for big data in recent years was caused by a combination of open source technologies to store and manipulate data and the increasing volume of data (Elliot, 2013). Add to that the various other 3rd platform technologies, of which big data analytics is part such as cloud computing, mobile and additional accelerators such as IoT and it becomes clear why big data gained far more than just some renewed attention but led to a broadening big data ecosystem (International Data Corporation, 2013). Today, we look at the business, intelligence, decision and value/opportunity perspective – from volume to value, and from chaos to mining and meaning – putting the emphasis on data analytics, insights and action.

2.2 Opportunities of Big Data

2.2.1. Big Data for Government

Easy and timely retrieval and analysis of related and unrelated information is crucial for government to meet and improve mission requirements that are varied across agencies. Data continues to be generated and digitally archived at increasing rates driven by Open Government initiatives, citizen interactions, and program transactions. Government organizations are beginning to deploy big data technologies to analyze massive data sets in science and research as well as mine data to prevent bad actors from committing acts of terror and/or to prevent waste, fraud, and abuse.

As government leaders strive to create data-driven organizations to successfully accomplish missions, chief information officers are laying the groundwork to correlate and track dependencies across people, processes, and information to ensure that the right information is available to the right person at the right time. Agencies need to take a strategic approach to where they are placing all this data and on the medium that provides the greatest success of access when needed. More departments, more functions, more use cases, more goals and especially more focus on creating value and smart actions and decisions. In the end it is what big data analytics – most digital transformation projects and enabling technologies such as artificial intelligence, IoT and so on – are all about (GetSmarter, 2018).

2.2.2. Big Data in Society

Understand the growing importance of big data in contemporary society and study the theory and practice of big data from an arts and humanities perspective. Beyond the large data sets that can be analyzed to reveal patterns, trends, and associations, it is increasingly about our everyday lives and how the data we generate is transforming social, cultural, political and economic processes and the generation of knowledge.

According to Qubole’s Report (2018), big data is being used across a wide and growing spectrum of departments and functions and business processes receiving most value from big data include customer service, information technology planning, sales, finance, resource planning, information technology issue response, marketing, human resource and workplace, and supply chain. In other words, pretty much all business processes. The shift to the cloud leads to an expansion of machine learning programs - machine learning is a field of artificial intelligence – in which enhancing cybersecurity, customer experience optimization and predictive maintenance, a top Industry 4.0 use case, stick out (Qubole’s Report. 2018).

2.2.3. Big Data for Policy Making

In recent decades, there has been a growing awareness that data can reduce uncertainty about the best course of action in policy design, i.e., that it can inform a better policy making process and lead to more adequate, more efficient and more effective public policies. Therefore, policymakers and policy advocates often tend to provide data based arguments for particular policy solutions, usually gained through sound empirical research or analysis on that topic (DataForPolicy, 2015). In that sense, big data technologies have created the opportunity for policy makers to have deeper, data driven, insights into respective issues and allowed quantitative analysis to penetrate the policy process more deeply than ever before. And this technological reality has generated the opportunity and necessity for a more complex, more sophisticated and technology driven approach to transforming data into policy action that goes beyond traditional empirical research.

Another important trend – political, rather than technical – that allows big data to penetrate into the public policy sphere is the opening of government data, namely, the growing demand for more transparent, accountable and responsive government, which is coming from citizens as well as from initiatives such as the Open Government Partnership, and resulting in more and more governments deciding to open up and make their data accessible (DataForPolicy, 2015). Illustratively, the US government has opened the door to its big data by launching data.gov and, through that, allowing stakeholders to approach, analyze or use the government’s data for various purposes. Once the data is open and available, opportunities to apply them in different spheres, from business to policy monitoring and analysis has become endless.

2.2.4. Big Data for Sustainable Development

The data revolution, which encompasses the open data movement, the rise of crowd-sourcing, new information communication technologies for data collection, and the explosion in the availability of big data, together with the emergence of artificial intelligence and IoT is already transforming society. Advances in computing and data science now make it possible to process and analyze big data in real time. New insights gleaned from such data mining can complement official statistics and survey data, adding depth and nuance to information on human behaviors and experiences. The integration of this new data with traditional data should produce high quality information that is more detailed, timely and relevant.

Today, in the private sector, analysis of big data is commonplace, with consumer profiling, personalized services, and predictive analysis being used for marketing, advertising and management. Similar techniques could be adopted to gain real time insights into people’s wellbeing and to target aid interventions to vulnerable groups. New sources of data such as satellite data, new technologies, and new analytical approaches, if applied responsibly, can enable more agile, efficient and evidence based decision making and can better measure progress on the Sustainable Development Goals (SDGs) in a way that is both inclusive and fair (Global Partnership for Sustainable Development Data, 2016).

2.3. Challenges of Big Data

2.3.1. Challenges to Adopting Big Data in Government

Government faces many challenges in managing the life-cycle of big data as government’s traditional silo approach hinders sharing knowledge and working across organizational boundaries. Government also processes multiple types of information (paper, digital, multimedia). IDC Government Insights research shows that, if paper documents were available in digital format, government workers would gain almost an hour a day. In addition, nearly half of government digital documents are printed, and almost 40% of these printed documents are reentered as digital documents (IDC Government Insights, 2012). But simply digitizing information isn’t enough. IDC Government Insights research also shows that a fourth of government employees were unable to find or access the digital information they needed more than half the time, indicating that making information available in digital formats is not enough and that records need to be indexed and properly tracked in order to be accessed.

Big data represents a big challenge for government executives given that today’s transaction based systems and silo programs discourage information sharing and hinder movement from process driven to data driven. Government leaders often have different drivers for using big data. For example, chief information officers need to support the Freedom of Information Act (FOIA), 5 U.S.C. § 552 in the United States, records management, and eDiscovery, all while driving down costs, adhering to policy and regulations, and supporting the use of big data by other agency executives and directorates.

Government executives recognize the importance of gaining insights through the proper storage, retrieval, and analysis of big data. The need of government decision makers to develop, deploy, and manage an effective big data environment is driving many to look to information storage providers to support their current and future needs. Tight budgets coupled with ever-growing government information and a move to digital information make it critical for government agencies to right-size their traditional and big data information storage. Many agencies are rethinking the roles of traditional storage and deploying a cost efficient and combined approach of tape for archival storage and cloud storage for big data information.

2.3.2. Challenges to Adopting Big Data in Society

Fundamental elements of human rights must be safeguarded to realize the opportunities presented by big data – privacy, ethics and respect for data sovereignty require us to assess the rights of individuals along with the benefits of the collective. Much new data is collected passively from the digital footprints people leave behind and from sensor enabled objects or is inferred via algorithms. Because big data is the product of unique patterns of behavior of individuals, removal of explicit personal information may not fully protect privacy. Combining multiple datasets may lead to the re-identification of individuals or groups of individuals, subjecting them to potential harms. Proper data protection measures must be put in place to prevent data misuse or mishandling.

There is also a risk of growing inequality and bias. Major gaps are already opening up between the data haves and have-nots. Without action, a whole new inequality frontier will split the world between those who know, and those who do not. Many people are excluded from the new world of data and information by language, poverty, lack of education, lack of technology infrastructure, remoteness or prejudice and discrimination. There is a broad range of actions needed, including building the capacities of all countries and particularly the least developed countries.

3. Case Studies and Applications of Big Data

The focus of big data in society should not only be on the extraordinary volume of information, but rather the value that organizations can extract from it. Using machinelearning technology, a field of big data analytics shows the value in large amounts of data. In a nutshell, big data analytics learns from data to predict the way individuals will behave in the future. Machine-learning detects patterns in data sets to consider the probability of certain outcomes. For example, the predictive model uses everything known about an individual to determine the likelihood of them buying a specific product, contracting a certain disease, being influenced by an economic trend, or any desired outcome (Siegal, 2018). While some may be skeptical about the implications of big data on privacy and security, big data stands to transform society for the better. We can discover its potential and application as follows (see Table 1 and 2 for more examples and applications):

Table 1: Big Data Technologies for Government and Public Policy

Table 2: Big Data Technologies for Smart Cities and Society

3.1. Government and Society

The first of the case studies is from India’s largest state, Madhya Pradesh, which is a low-income state with a large proportion of historically-excluded groups to showcase two data focused initiatives that have led to widespread transparency and improved service delivery. These are the Chief Minister’s dashboard that integrates information from various sources to promote evidence based decision making and the Chief Ministers Helpline, a unified gateway to register grievances related to all aspects of government functioning (Numanovic, 2017).

Barcelona developed plans to incorporate data into city planning over 10 years ago. The smart city it is hoping to achieve spans across various departments including water, energy, communication, housing, and mobility, all with the intention of improving people’s quality of life through technological innovation. The developments are underpinned with intelligent data systems collecting information from smart assets and public organizations with the intention of creating a more open, transparent government to engage citizens and provide greater public independence (Morabito, 2015).

The application of big data analytics in Western democratic governments is subject to much debate. While there are foreseeable benefits, democracy’s prioritization of privacy has the potential to hinder progress. Today, the big data revolution seems to play more into the hands of China’s leadership. With uninterrupted access to data through close collaboration with a few state-licensed commercial data conglomerates, i.e., Alibaba, Tencent or Baidu (Heilmann, 2017), the Chinese government could become a data powerhouse. However, while only time will tell how China’s approach to data will accelerate its technological advance, the rest of the world is by no means ignoring the potential application of big data in government sectors, despite the often strict privacy regulations.

3.2. Public Safety

Collaboration across countries and between public and private sectors may become more common as big data gains prominence. The Haiti earthquake in 2010 is an example of collaboration between businesses, emergency services, the government, and people in a disaster. To help the government, people, and emergency services cope, InStedd, a technology company specializing in emergency services, set up systems to decode data based on time, geolocation, and route of transmission. The quality of data is paramount in prioritizing information and minimizing incorrect usage of limited rescue resources. Together with various crowdsourcing agencies, they were able to accurately geotag information to provide coordinates to the search and rescue teams working on the ground, managing to save and support many more people (Apolitical, 2017).

Similar to disaster management, around the U.S. and 30 other countries across the world, Risk-Terrain modelling is being used to predict crime. Buildings are linked with types of crimes, and potentially dangerous areas are recorded on a digital map overlaid with past crime locations. Buildings are then ranked according to the number and type of crimes that have occurred close to their location and are flagged with police to alert them where to focus their efforts and attention (Apolitical, 2017).

The city government of San Francisco has employed a form of the big data approach to reduce frequent traffic collisions in the city. Namely, due to frequent traffic accidents, resulting in a number of deaths every year, as well as generally low safety in this regard, the Department of Public Health and the Department of Transportation were assigned by the government to develop an adequate policy to address this issue. They found out that just 12 percent of intersections result in 70% of major injuries. Finally, insights gained through this process were transformed into policy solutions, introducing ‘protected intersections’, underway intersections and protected bike lanes as some of the measures (Nemani, 2016).

3.3. Healthcare

By 2020, IBM predicts a 20% decrease in patient mortality (Kumar, 2016). As more data is collected and analyzed, it will be possible to save more people’s lives. With artificial intelligence able to understand questions, read through 200 million pages of data, and provide an answer in seconds, doctors can consider all available resources before making a decision regarding a patient’s condition (Kumar, 2016). In a consultation context, medical professionals will be able to analyze past trials, trends, and current data. With this information and access to data concerning a patient’s lifestyle, history, and genetics, a holistic picture can aid doctors in providing the most effective care. In much the same way businesses reach customers through targeted marketing, the goal in healthcare is to have enough personal data on each person to provide evidence based personalized treatment (Kumar, 2016). People would be empowered to play an active role in their health and wellbeing with information tailored to their specific needs.

Genome sequencing is another exciting possibility in the medical field. It took 10 years to decode the first human genome; today it takes a week (Kumar, 2016). As big data technology continues to expand and larger amounts can be processed, everyday genome sequencing for regular people may be possible. Today, there are only certain parts of the genome that are well understood and that influence our health care decisions. However, as more people are sequenced, scientists and doctors will have access to a larger set of data to learn about genes they previously couldn’t understand, including certain genomes relationships to diseases.

3.4. Policy-Making

The use of big data in policy-making is already a practice. Regardless of their political orientation, governments have extensively built their capacities to make choices based on big data inputs. A recent example of the use of big data for policy-making can be found in China. In April 2015, Rogier Creemers of Oxford University published a translation of “the regulations regarding the Chinese nationwide social credit system, attributing to each of its 1.3 billion citizens a score for his or her behavior. The system is to be based on various criteria, ranging from financial credibility and criminal record to social media behavior” (Creemers, 2016). The social credit system, a government-tailored policy made in cooperation with Chinese purchasing giants such as Alibaba, uses big data to assess the perceived trustworthiness of individuals, aimed to “strengthen social sincerity, stimulate mutual trust in society, and reducing social contradictions ...” (State Council of China, 2014). In practice, this policy may have direct effect on government behavior towards citizens.

In the United Kingdom, the government is currently implementing its Future Cities Catapult Initiative, which uses big data to help innovators turn ingenious ideas into working prototypes that can be tested in real urban settings and once they’re proven, help spread them to cities across the world to improve quality of life, strengthen economies and protect the environment. Similar initiatives are becoming increasingly popular throughout the European Union (EU), although Brussels firmly supports the Privacy Law (European Union, 2016), by which large international companies’ methods of obtaining data are put under scrutiny to some extent. The extent to which governments in EU use big data may hold potential to envisage the use of big data for making policies, which do not give governments exclusive insight into citizens’ lives (Gibbs, 2015).

4. Recommendations and Conclusion

4.1. Recommendations to Government

Government agencies should explore economically-viable automated records management technologies and solutions to reduce the burden of records management responsibilities. Technology is continuously evolving to provide advanced solutions to support data backup, recovery, and archiving requirements. Storage infrastructure that is capable of addressing the specific demands around data volume, velocity, and variety will be critical for government agencies to successfully use and process big data. Greater visibility into organizational information – together with the ability to produce documentation when needed – allows agencies to enhance employee efficiency and productivity, respond to questions, make informed decisions by connecting the related and unrelated information, and provide the information needed for transparency, collaboration, and participatory government.

Agencies should also strategically respond to the growth of data with plans that address the storage of traditional data as well as the storage of digital information – structured and unstructured and metadata. Budgetary constraints are forcing agencies to right-size their big data information storage. Plans should include the growing roles of tiered storage, tape, and cloud based backup and recovery as well as an analysis of the cost versus the benefit of multiple storage choices.

Cloud-based backup, recovery, and archiving provide different benefits and can play a complementary role in a storage strategy. For data that is constantly changing and requires short real time access, cloud technology provides an excellent backup and recovery solution. It enables movement of data offsite faster and more frequently and virtually instant retrieval. While cloud-based solutions can be more expensive than tape over the long term, they offer fast, automated backup and archiving that helps reduce information technology workload. Using cloud technology for backup, recovery, and archiving activities offers a flexible, scalable, pay-as-you-go approach to help keep costs down while minimizing resources required to manage related processes.

4.2. Recommendations to Society

2015, the world embarked on a new development agenda underpinned by the Sustainable Development Goals (SDGs). Achieving these goals requires integrated action on social, environmental and economic challenges, with a focus on inclusive, participatory development that leaves no one behind. Critical data for global, regional and national development policy making is still lacking. Many governments still do not have access to adequate data on their entire populations. This is particularly true for the poorest and most marginalized, the very people that leaders will need to focus on if they are to achieve zero extreme poverty and zero emissions by 2030, and to ‘leave no one behind’ in the process.

Big data can shed light on disparities in society that were previously hidden. For example, women and girls, who often work in the informal sector or at home, suffer social constraints on their mobility, and are marginalized in both private and public decision making. Much of the big data with the most potential to be used for public goods is collected by the private sector. As such, public-private partnerships are likely to become more widespread. The challenge will be ensuring they are sustainable over time, and that clear frameworks are in place to clarify roles and expectations on all sides.

4.3. Recommendations to Policy-Making

As the world is pacing towards data driven lifestyle, the global policymakers are showing a keen interest in hunting transformation in governance through big data technology. The changes brought in by the technology caters excellent opportunities for economic growth, productivity, innovation and various other concepts of governance and policy making. In particular, the policies designed under this umbrella are intended to collect national or international data under one efficient system of accessibility, accuracy, integration, and sophistication.

According to certain policymakers, big data is a cue to improve the implementation and effectiveness of government policies. Specifically, in the tax, healthcare and education and culture sector of government jurisdiction, big data is making remarkable impressions. Moreover, during the time of elections, big data extends noble hands to kindle winning results. Although it remains an under-addressed topic in government formation and country politics but from influencing voters to predicting the ultimate results of the election, big data knows it all.

After all, employing the technology is entirely about ways to improve the effectiveness and efficiency of national or international policies by accrediting governments across the globe to produce an informed policy for citizen welfare that can be analyzed in the real time schedule. Big data driven governance can equip nations to make better decisions without compromising the quality of it.

4.4. Public-Private Partnerships

With the advancements of new-age technologies fostering data and analytics marvels, the governments can process real time data and operate feedback cycle in every step of it. To make things run in hands of the technology, various entities of governments require to construct a system to gather large datasets and seek for new coils. Additionally, they also need to enhance the capabilities and outreach of their respective institutions to harness big data with talent and legit actions. Looking at the optimistic approach to this, the lack of data talent in particular country clears up room for technology partnerships in both the public and private sector. Both the sectors in every big data aspirant region needs leaders who are courageous enough to push the data literacy program at the helm. If big data could be underpinned with governments’ performance metrics, its implementation would be effective.

Every day, people around the world send hundreds of millions of tweets in dozens of languages. Such social conversations contain real-time information on many issues, including food costs, the availability of jobs, access to health care, quality of education, and reports of natural disasters. The partnership will allow UN development and humanitarian agencies to turn the public data into actionable information to aid communities around the globe.

Other examples of partnerships include the GSMA’s “Big Data for Social Good” initiative (GSMA, 2017), which leverages mobile operators’ big data capabilities to address humanitarian crises, including epidemics and natural disasters (Data for Climate Action, 2017); a competition which connected researchers around the world with data and tools from leading companies to enable data driven climate solutions (Data Collaboratives, 2017); and a new form of collaboration beyond the public-private partnership model, in which participants from different sectors  exchange their data to create public value.