Joulen accelerates the transition to sustainable energy practices for businesses using Polars

Many fast growing organizations and autonomous teams chose Polars for their data workflows. Main argument is that Polars easily scales with them without having to refactor or face a steep increase in compute costs.

Joulen had the opportunity to start from scratch and in this case study we can read how they set up their scalable data infrastructure. In addition, the case study shows how organizations can leverage Polars to enable everyone on the team to work on both small- and large-scale analysis using the same code base.

In an era where renewable energy and sustainable practices are becoming vital, and as businesses across the globe strive to achieve their net-zero goals, the importance of effective energy management cannot be overstated. In the rapidly evolving energy industry, startups like Joulen play a crucial role in facilitating the transition to renewable sources by providing sophisticated tools for energy management and trading.

The team at Joulen have developed the PARIS platform to solve this problem. PARIS is an AI-driven software solution designed to optimize renewable energy assets. It combines advanced forecasting, intelligent optimization, automated trading, and intuitive control interfaces to maximize the value and efficiency of renewable technologies. The platform empowers businesses to make data-driven decisions and accelerate their transition to sustainable energy practices.

The development team at Joulen requires a data analytics stack that is cost-effective and highly scalable. The three pillars of this approach at Joulen are; data storage in Parquet on S3, serverless computation with elastic containers, and data analysis with Polars. This approach is cost effective, not only from low cost storage on S3, but also because Polars uses query optimisations to limit the amount of data transferred. The approach is highly scalable as Polars uses all of the available computational power of the serverless containers. As the company grows, the team scales the pipelines by simply adjusting the computing resources of these containers.

A further advantage of Polars is that it allows the Joulen team to organize around a single data analytics technology. On a given day the team may need to do detailed analysis for a single client as well as run large-scale pipelines across all clients. As Polars works just as well on a laptop as on a large cloud instance the team can use the same code base for both tasks. Crucially, this also means that all members of the development team have the skills that allow them to work on both small- and large-scale analysis interchangeably.

The Joulen team uses Polars to extract analytics data from Postgres and KDB databases. This analytics data is stored in a lakehouse on S3 with the delta-rs library. The delta-rs library is an open source implementation of the DeltaLake protocol. The team chose delta-rs for its speed and tight integrations with Polars that allows the Polars query optimiser to optimize queries against the lakehouse on S3. The high performance of Polars and the lakehouse gives the team a real-time view of performance across the assets.

The core use case of Polars for Joulen is time series analysis. The team analyzes streams of data from solar panels, wind turbines and batteries to prepare the inputs for time series forecasting models and the battery management system. The team chose Polars for its fundamental speed and full set of features for time series analysis. Using machine learning libraries, such as XGBoost, Nixtla and Polars_OLS, the training and inference pipelines are fully run in Polars. In fact, the team even uses Polars plugins that allow linear models to be fit within a Polars expression!

Visualization is a key part of the analytics process at Joulen. The team built interactive Streamlit apps to generate insights into the performance of energy generation and storage assets. Under-the-hood these apps are powered by Polars as its processing speed gives a much better low latency experience for end users.

Energy management and trading is a complicated business. The key advantage of Polars is that it allows the data science team at Joulen to focus on optimizing our clients’ energy flows as Polars takes care of optimizing our data flows 

Liam Brannigan, Data Team Lead @ Joulen