Open Energy Data Initiative: SSEN's Digital Transformation (A technical case study)
Copy LinkDatopian
Scottish & Southern Electricity Networks
Brief summary of the project.
The Scottish and Southern Electricity Networks (SSEN) Data Portal project is an innovative digital transformation initiative aimed at enhancing power distribution management. Utilizing advanced technologies like PortalJS and CKAN, the portal effectively manages and shares large datasets, improving operational efficiency and user engagement.
SSEN faced challenges in efficiently managing, sharing, and accessing extensive data across its power distribution network, which is crucial for operational excellence and customer satisfaction.
There was a need for a robust platform capable of handling large volumes of data with high interoperability and security standards to improve data accessibility and streamline operations.
Datopian implemented a robust and comprehensive solution for SSEN by creating a decoupled architecture that utilizes PortalJS for the frontend to enhance user interface responsiveness and CKAN v2.10 for the backend to manage data effectively. This setup is augmented with several CKAN extensions such as ckanext-dcat for DCAT support to improve interoperability, ckanext-scheming for customized schema configurations, ckanext-showcase to highlight key datasets, ckanext-googleanalytics for in-depth usage analytics, and ckanext-noanonaccess to enhance security by restricting anonymous access. Additionally, an ETL workflow using Prefect and BigQuery facilitates the efficient handling and querying of large data sets, including smart meter data, ensuring that SSEN can manage data flows from over 3.8 million endpoints effectively. This solution not only improves operational efficiency but also aligns with sustainable energy management practices.
In our preceding case study, Empowering the Energy Sector: Utilizing CKAN and PortalJS to Build a Transparency and Innovation-Centric Data Portal for Scottish Southern Electricity Networks, we explored how SSEN is pioneering transparency and innovation by developing a data portal designed to democratize access to energy data. This initiative supports informed decision-making and accelerates the transition to net-zero emissions, aligning with SSEN's strategic goals of innovation and partnership within the energy sector.
This technical case study shifts focus to delve into the technical specifics of the solutions implemented. We will explore the architectural innovations and the integration of advanced data management technologies like CKAN and PortalJS that underpin SSEN's operational excellence and strategic objectives in energy efficiency and grid management.
Introduction
SSEN is a major power distribution company that powers millions of homes and facilities in England and Scotland. The company is responsible for managing the infrastructure, providing customer support, ensuring deliverability, and improving the performance of its power networks while focusing on sustainability and environmental footprint reduction. With more than 3.8 million endpoints, SSEN’s network is a major player in the European power industry.
Embarking on a visionary journey towards digital transformation, SSEN is reshaping the contours of power distribution management. At the forefront of this transformation is the implementation of an open data portal, along with the integration of smart meter data - a bold leap towards enhancing enterprise trust, operational excellence, and customer satisfaction. This digital leap is powered by PortalJS and CKAN v2.10, which were chosen for their robustness and adaptability. This case study offers an in-depth exploration of the architectural innovation behind this transformation and the impactful outcomes that are redefining industry standards.
Architecture & Used Technologies
The following diagram depicts the general architecture of the SSEN Data Portal, providing an overview of its structural service components and their interconnections. The core of Data Portal's architecture features a robust data management and sharing system that facilitates seamless data storage and sharing with an excellent user experience.
The architecture of the Data Portal is organized into distinct services, each tailored to serve a specific purpose and contribute to the overall functionality of the platform.
1. Decoupled PortalJS Front end:
The front end of the Data Portal is implemented using PortalJS, a powerful and modular JavaScript framework built on NextJS. This strategic decoupling of the front end from the backend infrastructure has enabled the Data Portal to achieve a user interface that is not only highly responsive but also aligns with modern design principles. Such an architecture facilitates independent development and scaling of the frontend components, which is crucial for maintaining a fluid and captivating user experience.
PortalJS stands out for its adeptness in managing complex user interactions, data visualization, and the presentation aspects of the portal. The framework's design is inherently focused on optimizing performance and user engagement, ensuring that the portal remains intuitive and efficient in handling diverse user requirements. This approach to front-end development reflects a deep understanding of the needs of contemporary digital interfaces, placing user experience at the forefront of the portal's design ethos.
2. CKAN DMS Backend
The backbone of the Data Portal is powered by CKAN, a robust open-source data management system. This CKAN admin backend acts as the central administrative platform for publishing and organizing datasets. It provides essential backend functionalities to perform tasks such as data storage, retrieval, data metadata management, and user role management, ensuring that data management features are seamlessly integrated into the portal.
With the installation of the following extensions, CKAN further extends its capabilities, enabling additional features and improving the platform's functionality.
This extension enables interoperability by implementing support for DCAT (Data Catalog Vocabulary), a standard schema for describing datasets. The integration of ckanext-dcat, significantly enhances the Data Portal's compatibility with open data initiatives, facilitating seamless data sharing and integration with other platforms that adhere to the DCAT standard.
In addition to DCAT, the portal also requires interoperability with Dublin Core and Icebreaker One metadata standards. To address this requirement, several custom modifications have been implemented. These custom enhancements play an essential role in facilitating the Data Portal's seamless integration within diverse data ecosystems. Moreover, they are instrumental in aligning the data portal with specific standards that are critical to the operational frameworks of SSEN. This way the data portal remains versatile, adaptable, and in sync with evolving industry standards and practices.
The implementation of the Ckanext-scheming extension is a key facet in the customization of the Data Portal's schema, tailored to meet specific operational needs. This extension allows for a high degree of flexibility and precision in schema configuration, ensuring that the data portal aligns seamlessly with the unique requirements of SSEN's metadata framework.
Ckanext-showcase offers features that improve the presentation of high-value datasets in the Data Portal. This extension allows administrators to create curated showcases, which serve to highlight key datasets, ensuring that important information is prominently displayed and easily accessible to users. This feature enhances the data portal's functionality by making critical data more visible and user-friendly.
This extension integrates Google Analytics into the Data Portal, offering valuable insights into user behavior and engagement. This extension equips admins with comprehensive analytics, enabling them to understand how users interact with the platform in depth, allowing for data-driven improvements and optimizations to enhance the overall user experience.
This extension enhances the security of the SSEN Data Portal by restricting anonymous access to the CKAN classic front end. Users are required to authenticate, redirecting unauthenticated visitors to the login page. Access is only permitted once users are logged in. This functionality is particularly pertinent for the SSEN Data Portal given its decoupled frontend architecture. While general users typically engage with the portal for data consumption and do not require access to the admin UI, this extension is essential for administrators and data publishers. It allows them to securely log in and perform critical administrative tasks such as dataset publication and user management, thereby maintaining the integrity and controlled accessibility of the Data Portal.
3. ETL Data ingestion workflow
SSEN also publishes meter data through the portal, a task that entails managing significant amounts of data. The accumulated volume of metadata is substantial, often exceeding approximately 1 TB. Handling such a large dataset, while providing users with the ability to access specific subsets of data, poses a considerable challenge for conventional PostgreSQL relational databases.
As a solution, Datopian has implemented a Prefect-based ETL (Extract, Transform, Load) workflow. This process begins with SSEN's daily upload of meter data, formatted as a flat CSV file, into a blob storage bucket. The ETL workflow then takes over, retrieving this file and efficiently uploading its contents to BigQuery. The use of BigQuery, known for its robust cloud-based infrastructure, allows for the swift and flexible querying of vast data records. Additionally, Prefect plays a pivotal role in streamlining the workflow, ensuring seamless orchestration and comprehensive management of the entire ETL process.
Results
The successful implementation of the SSEN Data Portal marks a significant milestone in the company's digital transformation journey. Merging the sleek, user-centric design of PortalJS for its front end with the robust, data-intensive capabilities of CKAN v2.10 as the backbone, the platform achieves a synergy between seamless, responsive user interface and powerful data management and sharing capabilities.
This strategic development effectively addresses the challenges of handling and querying large datasets, a common concern in power distribution networks. The integration of BigQuery stands out in this context, offering swift and flexible data analysis solutions. As a result, the SSEN Data Portal stands as a prime example of technological advancement, efficiency, and sustainability in power distribution information management.
By prioritizing innovation and user experience, the SSEN Data Portal not only enhances operational efficiency but also aligns with sustainable energy management practices. This approach positions SSEN as a leader in the digital transformation of the energy sector. It is a clear reflection of the company's forward-thinking approach, setting a new benchmark for the industry and paving the way for future digital solutions in power distribution and energy management.
Next Chapter: Unveiling the Power of Smart Metering in SSEN's Digital Transformation
In our next case study, we will focus on one of the most transformative elements within SSEN's digital portfolio: smart metering. We'll delve into how SSEN, with Datopian's technological support, has become the first Distribution Network Operator in the UK to publish comprehensive Smart Meter half-hourly consumption datasets. This initiative involves aggregating data from over 1.8 million smart meters, yielding over 170 million daily readings, which are made accessible through SSEN's Open Data Portal. The data offers critical insights into 84,000 Low Voltage street-level feeders and over 36,000 distribution substations. This isn't just big data. This is huge data. With Datopian's expertise, this Data Portal is poised to grow, housing 1-2 billion rows of data within a year, updated every 24 hours – a pioneering step in data accessibility worldwide.
The upcoming case study will highlight how this data is leveraged through SSEN's Open Data Portal to support the UK's net zero transition, enabling precise modeling of electricity consumption and facilitating the development of low-carbon solutions. Stay tuned to uncover how this ambitious project is setting new benchmarks in data accessibility and contributing to a sustainable future.
