For an API to be successful, it needs to meet certain quality criteria, such as:

• intuitiveness, developer friendly, focused towards easy adoption
• stability and backward compatibility
• security
• very well documented with up to date documentation

Focusing on the last one raises a few questions:

• How do we document?
• How do we make the documentation public?
• How can we keep the API and the documentation in sync in real-time with minimal maintenance effort?

The sky is the limit here, but on the other hand there is also a clear standard to describe APIs which is OpenAPI Specifications.

For our services, the spec is automatically generated from the C# XML doc comments.

This will output a JSON or YAML standardised specification that can be interpreted by any other tool higher up the stack that supports the same standard. The ubiquitous swagger UI is the most well known example.

That is great for local development and internal access, but once the API goes live we need something publicly accessible by any consumer or by our partners.

This is usually done by uploading the specs in some kind of developer portal that supports OpenAPI Specs and offers support features like partner login, public access, high availability, user friendly UI, playground to try the API etc.

While there are a few examples and solutions, we will focus on ReadMe.io which basically takes as input an OpenAPI spec and layers a nice UI on top of it.

The goal is, once a developer makes a change to the API, as soon as the code is deployed to production and ready to be consumed, the documentation magically updates at the same time without additional work. No copying and pasting of text, no manual editing, no updating of wiki pages etc.

The immediate go-to solution is to automate this process in the CI/CD pipeline.

What we need:

• tooling to push the file in readme.io – this is provided by readme
• an API spec file (json/yaml) – as input to the readme CLI
• automation in the CD Azure pipeline – to call the CLI whenever we have a code update

Here are the Azure pipeline tasks that do the trick:

There are a few important points here that can be solved in other ways depending on your setup.

You might run into the case where your API deploys inside some infrastructure where Azure DevOps pipelines do not have access. This was our case:

We built our custom API gateway using YARP (which isn’t very compatible with swagger).

You can work around such a problem by exposing the swagger endpoint in the API gateway.

If you have multiple services, you expose them on different endpoints in the API gateway. YARP requires unique URLs for different endpoints.

For example:

• [https://$](<https://$/>)(API_GATEWAY_CUSTOM_DOMAIN_HOSTNAME)/swagger/myAPI1/v1/swagger.json
• [https://$](<https://$/>)(API_GATEWAY_CUSTOM_DOMAIN_HOSTNAME)/swagger/myAPI2/v1/swagger.json

Then implement a URL transformer in YARP to rewrite the URL to /swagger/v1/swagger.json just before forwarding the requests to the private services.

This is enough for most scenarios, but there is yet another special case that can occur on top of everything else

In case you have multiple APIs, you might keep the input/output models of that API in a separate nuget package that is referenced by all APIs.

In this scenario, the generated OpenAPI Spec will miss all the XML documentation that is part of the external nuget.

All the underlined lines will be missing by default!

The problem is that there is a bug in the .NET project system that prevents from copying the existing XML doc file located in the nuget package in the output folder of the API.

1. A bit of manual intervention is required inside the .csproj file:

This will iterate over all dll files and will try to copy the respective XML files to the output directory, if the file exists and if it matches the name of our models’ nuget. Otherwise a lot more XML will be copied over and will needlessly increase the deployment size.

This will work beautifully locally, but it will not work in a docker environment where the image is built by Azure pipelines.

2. To make it work from the pipelines, where you will likely use dotnet publish command, a new line is to be added after the initial copy operation.

The only difference will be the destination folder, instead of the output path it will be the publish directory.

3. The last touch is to add the NUGET_XMLDOC_MODE environment variable to instruct the restore operation to unpack the XML docs from the nuget packages together with the DLLs. This is set by default to skip to shave a few seconds off the pipeline run time.

To ensure the success of your API, don’t neglect the documentation side of it. Never rely on manual updates of wiki pages, Confluence pages or even using the UI editors that some tools and services put at your disposal.

Always strive to automate and correlate the API code deployment with updated docs deployment.

Depending on each particular case, the system and technology stack used and the tactics used, the hoops you might need to jump through to achieve this can vary – but the strategic goal should be the same.

Nowadays, OpenAPI Specifications are the de facto standard for describing and documenting APIs, enabling easy interoperability between API providers and API clients and consumers.

Producing a high quality OpenAPI spec will have a significant impact on the success of your API as it is the basis for a lot of tools that sit higher in the stack like:

• swagger UI
• developer portals
• autogenerated API clients (language agnostic)

In the .NET world, the most common way to generate an API spec is by using swagger. When properly configured, swagger supports versioning and automatic documentation based on the autogenerated XML docs.

Hence the better and more complete the XML docs, the better the resulting documentation and specification of the API.

In the next sections, we will be putting it all together from a 0 to a hero API spec that is:

• versioned
• serves multiple audiences
• well documented
• automation ready

Enable Versioning

The first step is fairly simple. We enable versioning and configure swagger:

Line 11 to 21 will pick up on all XML doc files that are found and loaded into swagger.

Line 9 is a placeholder for features detailed in the next section.

Customising Swagger Generation

There are times when you want to make programmatic changes to the way the final OpenAPI spec json file is generated. For example, exposing in the OpenAPI spec an HTTP header that is used by all endpoints, which in turn will show up on the UI to be easily filled out by the user.

For this, we can use swagger filters. Think of filters like .NET middleware but for swagger.

Document filter applies to the whole document while the operation filters are applied for each endpoint path.

Everything so far can be nicely packaged in a shared nuget library to be reused across multiple API services.

Multiple API Audiences

It could often be the case where you might have in the same API service both public endpoints that are exposed to partners via a gateway and private, internal ones that are only used by your team to carry out admin tasks or what not.

The internal endpoints will be part of the same logical API version, ex: v1, v2, v3 etc. but should be hidden from the public spec that the partners really see, and on the other hand still be visible in the internal swagger UI where your team (developers) can have access.

Considering the infrastructure pieces from previous sections are in place there are only two things left to do:

• define internal versions of the API

Strictly speaking there are two asp.net versions defined but only one logical version (1.0). If you leave out one version, that endpoint will simply not appear in the generated spec for that asp.net version.

This will generate a swagger UI that will allow us to pick which definition to use:

Fine Tuning

Usually you will need to fine tune the specs based on the audience. For example, if we want to expose the spec in a developer portal that will probably have a different host server than the localhost version in swagger UI. The ideal place to make these conditional tweaks is in swagger filters.

For instance, the AddServerFilter from the previous section can look something like this:

This will add the following attribute in the resulting OpenAPI spec, that will be later picked up by developer portal tooling and UI.

Mapping XML Docs to Swagger UI

The quality of the resulting OpenAPI spec is determined from the quality on the XML docs, attributes and signature and data types of the API endpoints. The more complete they are, the easier it is to use swagger UI or any other similar tool based on OpenAPI specifications.

Take the following code as a source:

This will generate the following swagger UI. Notice the entityId example and default value for type. They are both pre populated and ready to use. Notice the remarks section which is totally optional and not added by default. Once present, it will give the API endpoint a more detailed description.

Other tools provide similar capabilities and will offer the clients of the API a greatly improved experience.

Using the same XML docs in the models that are used as part of the endpoint interface, we obtain a nicely documented schema. Notice all the descriptions in the response schema below:

This article, written by re.alto’s development team, explains how to scale with Azure Container Apps and Apache Kafka. While such documentation already exists for use with Microsoft products, our development team did not find any similar documentation on how to scale containers using Apache Kafka and Azure Container Apps. We have since figured out how to do this ourselves and want to share our knowledge with other developers. The article below is intended to act as a guide for other developers looking to do something similar.

(For an introduction to this topic, please see our previous article on containerisation and container apps here.)

To allow containers, specifically replicas of a container, to be scaled up and down depending on the number of messages flowing through an Apache Kafka topic, it is possible to set up scaling rules for the container. But first, we need to create a container image that can consume messages from an Apache Kafka topic. 

Consume message from an Apache Kafka topic:

To consume messages from an Apache Kafka topic, various solutions are already available. Since we have chosen to host our Apache Kafka cluster with Confluent, we have also decided to use their nuget package. You can find it on the nuget.org feed under the name Confluent.Kafka.

When you install this nuget package, you can use the ConsumerBuilder class to create a new consumer and subscribe to a topic. 

Now all that is left is to consume the messages.

Now you have the code to consume messages from an Apache Kafka topic. If you would like more information, you can have a look at the documentation provided by Confluent at https://docs.confluent.io/kafka-clients/dotnet/current/overview.html. Now we build a container image with this code inside, register it at our container registry and create a container app based on the image – but how do we tell Azure Container Apps how to scale this container?

Scaling Azure Container Apps:

Azure Container Apps uses a KEDA scaler to handle the scaling of any container. You will not have to configure the KEDA scaler yourself, but you will have to tell the container which scaling rules to use. You will find a lot of examples on the Microsoft documentation pages on how to scale based on HTTP requests or messages in an Azure Service Bus. However, if you would like to know how to configure the scaling rules for an Apache Kafka topic, you may find yourself out of luck with the available documentation. But we explored and managed to do it like this:

You will need to set up a custom scaling rule for your container. We are using bicep to deploy our containers, therefore the examples shown below will be in bicep, but they will translate easily to any other way of deploying to Azure. To understand bicep or have a look how to create a bicep file for an Azure Container App have a look here: https://learn.microsoft.com/en-us/azure/templates/microsoft.app/containerapps?pivots=deployment-language-bicep.  

We are going to focus on the scaling part of the container, so to begin with, you need to configure the minimum and maximum number of replicas that the container can scale between.  

In the example above, we have set the minimum number of replicas running to 0, which means it will scale down to zero replicas running when there are no messages to consume from the Apache Kafka topic. This means that we can save on costs, as well as freeing up those resources for something else. 
The maximum number of replicas is set to 6. You can go as high as you like, but going above the number of partitions in the Apache Kafka topic would be pointless, since any replica above the number of partitions will not be able to consume messages. So this number should be set to any number that is less than or equal to the number of partitions of the Apace Kafka topic. 

Now let’s add the rule. 

The name of the rule can be anything you like; the name chosen here is just an example. After configuring the name, we need to define the rule. In our case, it is a custom rule. The first thing that the custom rule needs to know is the type of the rule. This translates to the KEDA scaler that is going to be used for the rule. We want to use the Apache Kafka KEDA scaler, but any available KEDA scaler can be used here.  

Next comes the metadata, which means we need to provide the bootstrap servers, the consumer group and the topic that was used in the code sample to inform the KEDA scaler which Apache Kafka topic to listen to and determine whether scaling up or down is needed.  

And as a final step, we need to allow the KEDA scaler to access the Apache Kafka topic. We have created a few secrets to store the required information (please look at the Microsoft documentation regarding deploying Azure Container Apps mentioned above for more information). We will have to provide the bootstrap servers, the username and password and the type of authentication to the KEDA scaler to allow it to connect to the topic. Once this is all configured and the container is deployed using the bicep module, you have your Azure Container App configured to scale based on the number of messages in the Apache Kafka topic. 

Containerisation: An Introduction to Technologies Used in Containerisation

& a re.alto Use Case Example 

In this article, we want to focus on the more technical side of re.alto and are going to highlight some of the technologies currently used by our developers when it comes to container apps.   

Containerisation is the bundling of software code with all its necessary components into one single, easily portable package (a “container”). It allows software developers to deploy and scale applications more efficiently and removes the need for running a full operating system for individual applications.  

Docker, an open-source platform for developing software in containers (applications and any supporting components), enables developers to separate their applications from their infrastructure. Developers write code to create a container image, this image is then deployed to a container image repository. A container is based on a container image – when the container starts up, it will pull the image and execute the code inside it. Depending on the number of requests (ie: calls to an API), a container can have zero to a lot of replicas (instances of a container) running simultaneously, in which case an orchestration tool such as Kubernetes is often required to handle this. 

Kubernetes is an open-source platform that manages containerised applications and services as clusters. Kubernetes offers a framework for software developers to run distributed systems efficiently, handling the scaling of applications and providing other useful features, such as load balancing and self-repair of containers. However, it can be complicated to manage your own Kubernetes cluster without prior experience of doing so. Most developers lack interest and knowledge in this area as it is expertise that falls more within an infrastructural role in a company – and many start-ups have not yet scaled to the size where this hire is necessary. That is where Microsoft Azure Container apps comes in. 

Like many smaller companies (and especially start-ups), re.alto does not have the time or resources to manage our own Kubernetes cluster, so we decided to use Azure Container Apps, an orchestration service introduced by Microsoft in 2022 for deploying containerised applications. This service really is a game-changer for start-ups, as it greatly simplifies the management of a Kubernetes cluster. Developers can still create containers but no longer face the hassle of configuring and maintaining their own cluster, as Microsoft does all this for them. With Microsoft’s managed environment taking over the orchestration of their cluster, developers can focus more on the actual container apps they want to build. 

How we use containerisation (Azure Container Apps) and Apache Kafka to get data from a smart meter dongle and stream it elsewhere. 

One use case, for example, focuses on the flow of our Xenn P1 dongles. The image above shows the stages of this flow. In this case, the IoT device – our Xenn dongle (attached to a smart meter for use with re.alto’s Xenn app) – pushes a message to an MQTT broker. A container consumes this message. The raw message is then distributed by Apache Kafka. Kafka has the ability to listen to millions of messages per second and has the benefit of storing messages for a certain period of time (in our case, seven days). That means if the connection to one container is temporarily lost, or if we stop and restart it, it will still read and process any messages from that period once it is back online – meaning no messages/commands are missed or lost. Kafka keeps track of which consumer groups (or container apps) have read which messages and, although we don’t do this currently, it is also possible to set up a schema for an Apache Kafka topic which only allows you to push messages in a specific format into a topic, guaranteeing that each message is of a specific format/standard. Using Kafka is greatly beneficial to companies like re.alto as we have far too many data streams to manage all of them ourselves. The message is then picked up by another container app and is stored in our data storage. While all containers receive the same message, they can be programmed to follow different instructions. This enables us to have containers with individual responsibilities, such as a container for tracking peak detection. In this case, the container would extract the relevant parts of the message and compare the readings with all other readings in the same quarter-of-an-hour to determine whether the consumer will run into a peak. If this is the case, the consumer receives a push notification in the Xenn app.  

Azure Container Apps provides start-ups with an invaluable service in the management of Kubernetes clusters, while Kafka simplifies handling large amounts of data streams and ensures no messages from IoT devices get lost or go unread. 

In our next article, we’ll be looking in more technical detail at how to scale with Kafka and Azure Container Apps and will provide you with some of the code needed to do so. 

Web APIs are becoming increasingly widespread in the energy sector. Historically they have been a popular technology for communicating market data (e.g. wholesale electricity prices) and grid information. Now we are seeing more and more parties adopting APIs as part of their digitalisation strategy.

This tutorial is for anyone that wants to get a better understanding of what an API does and how to build one from scratch in under an hour- all in the context of the energy sector.

For this tutorial, we are going to pretend that we are a market intelligence company that is researching trends relating to Power Purchase Agreements (PPAs). Our goal is to create and maintain an internal database of PPA deals which we can interact with via an API. The API will be simple but allows us to Create, Read, Update, and Delete data (typically referred to as a CRUD application).

To develop our API, we the going to use a popular web development framework called Express.

Express is described as “a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications”. Node.js is a JavaScript runtime environment built on top of Chromium’s V8 engine. It essentially enables you to run JavaScript code outside of a web browser (e.g. for building scripts, backends, APIs, etc).

There are tons of other web development frameworks that can be used, some examples include:

• Koa.js, Meteor.js (JavaScript)
• ASP.NET Core (C#)
• Django, Flask (Python)
• Laravel (PHP)
• Spring (Java)

For this tutorial we’re going to use Express since it allows us to get up and running very, very quickly 👍

You can find all of the code we will be writing for this project on our GitHub page.

Before we start, we’re going to need a couple of things:

• Node.js installed on your computer
• Some kind of code editor (optional) – a popular option is VSCode
• A command-line-interface to run our application – on Windows, Command Prompt will do

Go ahead and install Node.js. Once done, let’s begin by setting up our project.

1. Open up a Command Line Interface (CLI) such as Windows Terminal, PowerShell, Command Prompt, Bash, etc.
2. Browse to wherever you want to store your project code (I use C://src or C://www for all my projects).
3. Create a new directory for the project named “build-api-tutorial” and lets go into the root of our new directory.

In Windows Terminal / PowerShell, I’m using the command “cd PATH” to browse directories and creating a new directory using “mkdir NAME”. You don’t have to do this in a CLI of course, just create a new folder using your standard file browser. Here’s a list of CLI commands you might find useful.

Using your CLI, go to the root folder of the project. Let’s run the command “npm init”. NPM (aka node package manager) ships with Node.js out of the box and allows us to pull different packages.

You’ll be prompted to enter info relating to the project. For this tutorial we can stick with leaving things as is, hit enter/return until the prompts have cleared.

Great, you should now see a new file called package.json in the folder. The last thing we need to do before getting to the code is install express. Run the command “npm install express” in your CLI. Once installed, you should be greeted with something like:

Next create a blank “index.js” file and save it to this directory from your code editor. Your folder structure should now look like this:

“node_modules” contains all of our external packages that we installed (including Express and its dependencies). “package-lock.json” contains a list of our dependencies and the versions locked to our project. With set-up out of the way, it’s time to get started!

To begin, we’re going to use the “Hello World” example described on the Express website. Copy and paste the following snippet into your index.js file and hit save.

Go back to the CLI and type “node index.js”. This tells Node.js to run the index file containing our Express API server. You should see the following message which lets us know that the server is running locally:

Open up a browser, go to “localhost:3000” and you should be greeted with the following message:

Awesome! You have just built your first web API in 11 lines of code 😎

The client/browser requests an endpoint (http://localhost:3000/), the API server receives the request, and returns a message, in this case “Hello World!”.

You can shut down the server by opening up your CLI and holding the “Ctrl” key and pressing “C”. Make sure you restart the server before trying to run any updated code.

As we are going to build an API dealing with PPA deals, it’s important to define what that data actually looks like.

PPA deals typically have the following in common:

• A buyer and seller of electricity
• The technology used (e.g. solar/wind) and capacity of the installation
• The duration of the contract and the country and date where a deal took place

Working with a database is outside the scope of this tutorial, so what we are going to do instead is mock one. We are going to take the key bits of information above and store the data in memory as JavaScript objects like so:

The object above is comprised of a “key” – e.g. “buyer”, “seller”, etc. And its corresponding “value” – Company Y”, “Company X”, etc.

An aside: When using actual databases, storing data as objects with key-value pairs like this exist. They are typically referred to as NoSQL-type solutions. Of course there is many different paradigms and ways of storing data. One of the most common being SQL-type databases which share a lot of similarities to the humble spreadsheet. In this type of database solution, columns represent attributes (e.g. seller, buyer, country) and rows represent each record with a corresponding set of values.

Cool, now we have a structure for how we want to store and represent data. Let’s implement it on our API.

What we will do first is mock a database by creating a variable which stores an array (or list) of our PPA deals. Each deal stored in our array will have the object structure from above.

Open up “index.js” and edit your code to the following:

Hit save. Then let’s fire up the API by typing “node index.js” in your CLI. You should see something like this:

Here we are just logging/printing the data to screen to check everything is working as expected. With our “database” created, it’s time to move on to the fun part.

In designing our API, we’re going to follow RESTful standards. Microsoft have a great article on the topic if you want to learn more. For our purposes, the API we want to build has the following functionality:

• Return a list of PPA deals, which can optionally be filtered
• Get details about a specific PPA deal
• Create a new PPA deal
• Update an existing PPA deal
• Delete a PPA deal

Translating these requirements into API routes (or endpoints) using a REST approach would yield the following logic:

Above we’re using the same function used in the “Hello World” example to define the different endpoints of our API.

These endpoints represent specific URLs that we can call on our API with various HTTP methods (e.g. GET, POST, PUT, PATCH, DELETE).

Depending on how we call the endpoint, and what variables we pass through with our request, allows the server to respond in different ways.

Let’s look at an example – update your code to the following:

Save and and run the server (CLI -> “node index.js”). In a browser go to “localhost:3000” and you should be greeted with a familiar message. When you go to “localhost:3000/api/deals” however you should see this:

The API receives a GET request from the browser for the endpoint “/api/deals” and responds with the PPA data we defined earlier. Protip: the above might look like a mess on your browser depending on if you have a JSON formatter extension installed or not. We’re going to be using a dedicated API tool later in the tutorial so don’t worry about it if that’s the case.

Now that we have defined what we expect our API to do and the corresponding routes, it’s time to flesh out the business logic.

In general, when an API receives a request, the following takes place before an action is performed (not necessarily in this order):

• Routing: Did the request hit a valid/existing endpoint with the correct method (GET, POST, etc)? If yes, carry on, otherwise return an error.
• Auth: Is the endpoint “protected” in any way? For example, the endpoint might only be allowed for certain users to access such as when deleting data. The API checks that the request has valid permissions to perform the action (authenticated/authorising the request). Unauthorised requests are rejected.
• Validation:  Is the endpoint expecting any variables or a payload? If they are missing or erroneous (e.g. wrong format, data type, etc) the API returns an error.
• Security: Is the request compliant with all security measures? For example are the requests adhering to the rate at which the API is allowed to be called (e.g. 100 requests / minute).

Once a request has gone through the checks above, our server is then free to get down to business and deal with it.

Let’s flesh out our API by adding some basic validation logic and business logic (which deals with the “database” itself) to the endpoints described previously. Note, the coding itself is outside the scope of this article but if you would like to learn more about coding then I highly recommend freeCodeCamp as a resource. Copy and paste the final bit of code from below (or alternatively from our GitHub page) into “index.js”.

You will notice that there’s a fair amount of additional code! In addition to the “database” and routes we defined earlier, you will notice a bunch of new functions (services) which perform concrete actions such as validating an input or making a change to the database. Near the bottom are the routes we defined earlier, and these encapsulate the logic of how our API works. In production-grade code, this “controller” logic would typically be separated from the routes themselves – but for demo purposes this works well.

As a walkthrough example: If we look at the the “Create a new PPA deal” endpoint you can see that “/api/deals” accepts a POST request. You can see that our endpoint is expecting to receive a body of data from that request, which it is then validating using the “validateData” service. This service will check that all keys expected for the data exist (e.g. buyer, seller, technology, etc.). If it doesn’t, then the API returns an error. If the body of data is valid, then we add an entry to the database using the “createDeal” service. If that is successful, then the API responds with a message confirming that the data was inserted.

There’s a number of comments included to try and explain what each piece of code is doing. If anything is unclear please feel free to reach out or comment.

With our API built, it’s time to get testing! Boot up the server one last time (CLI -> “node index.js”). Next, we’re going to grab a popular API testing and development tool called Postman. Postman will make it super easy to test and play with our API, so head over to Postman to create an account. Once logged in, let’s create an API request.

On the next screen, add “localhost:3000/api/deals” in the URL field and hit “Send”. You should see the data we defined earlier.

Great, that route seems to work! Let’s try something new, update the URL to the following and hit “Send”:

See what happened? You’ll notice that only three PPA deals were returned this time (versus the five we originally described). We have asked our API to process any “query parameters” included the URL and use these to filter the database against. The three records shown are now filtered to PPA deals using solar technology. Lets try adding another query parameter, copy and paste the following URL:

Now we only receive a single data object. A PPA deal originating in the UK with solar as the technology. This type of filter logic helps applications which deal with a large amount of records and/or fields. You can imagine that we might create a front-end app or BI dashboard which connects to this API and is able to filter results interactively.

Let’s try out some of the other routes. Copy the following and hit “Send”:

You will notice that only a single record which has an “id” of 3 is returned. The above is called a “route” or “template” parameter. The variable itself is included as part of the URL directly, and not constructed like a query parameter – i.e. “api/endpoint?key=value”. Let’s try a new method, this time, instead of a GET request to the URL above, let’s send a DELETE request.

Here we are instructing our API to delete the PPA record with a unique identifier equal to 3. Hit “Send”. If you call the “/api/deals” endpoint with a GET request again, you should see that only four records are now shown. We have successfully deleted a record from our database.

How about creating a new record? Set-up a POST request and aim it at the following route:

If you hit send now, you should see this error message:

When we’re calling this particular route, the API is expecting a request “body”. The body contains the data with which we want to create a PPA deal for. Go ahead and copy the following snippet:

Click on the “Body” tab, click on “raw”, then update the data type to “JSON”. Paste the snippet in the box below.

When we send this request, our API is going to validate the data received, and create a new record if all checks pass. If all good, the API will respond with the new record that was created. Hit “Send” and you should receive a response with the newly created record, and a corresponding “id” field for the record. We’ve just successfully inserted data into our database, nice!

Updating a record works in much the same way as creating one, you just need to send a PATCH request to the “/api/deals/{id}” endpoint. By including “id” in the route, you’re specifying which record you would like to update. This endpoint also expects a body object similar to the one above. Give it a try!

And that concludes the testing phase. There’s a lot of other things we can do to test and improve the API, but as an introduction to the technology, we’ve covered a lot of ground.

And there you have it! In this tutorial we defined, built, and tested an API from scratch using our PPA use-case as an example. I hope the tutorial was useful in explaining a bit more about how they work and also in showcasing the power of APIs for both internal and external use. All of the code shown in this tutorial is accessible over at the re.alto Github page. Happy coding!

Web APIs are becoming an increasingly popular standard for communicating data. The days of CSV downloads, file transfers, and even web-scraping are numbered, and for good reason. Before APIs can be used and integrated into any system or process, however, there’s usually an element of investigation and play that needs to happen.

Understanding how an API works, what data it contains, how the data is structured and formatted are all activities which have to take place upfront. One technique which helps with this is the process of collecting and transforming data into a state which is ready to be visualised. That’s what we’ll cover today 😎

For this tutorial, we’re going to create a Python script which pulls Belgian solar forecast data via an API, post-processes it into a Pandas dataframe and finally visualises the output into a time series graph.

Why solar forecasts? Well solar forecasts are an interesting time-based dataset to work with since they’re critical for many different actors within the energy sector. For example grid operators, renewable asset owners/operators/developers, traders, smart buildings, energy management systems, the list goes on and on.

So let’s begin!

What you’re going to need

• A Python IDE (I personally use Spyder)
• Postman – a simple-to-use app which allows us to interact with APIs easily
• An account on re.alto – an API marketplace for the energy sector
• A subscription to Elia’s solar forecasting API

Querying the Elia solar forecast API using Postman

Before we get into any coding, let’s first take a look at the API which Elia (the Belgian Transmission System Operator) has created. On the technical operations page you can see that there’s only one operation/endpoint to integrate and a variety of parameters we can play with.

Let’s boot up Postman and create a request using the following query parameters:

We also need to include an API subscription key (retrieved under the “My Subscriptions” area on the re.alto portal). This needs to be included in your request header with the following key-value pair:

“OCP-Apim-Subscription-Key” : “YOUR-API-TOKEN“

Hit run and your Postman client should get a response. For example:

Okay great, so now we have a fairly good idea of what data the API contains and also what post-processing work we’ll need to be done before we can visualise the output.

Over to our Python script, let’s start of by defining our imports and variables.

## script.py

# imports

import requests
import pandas as pd
import re
import matplotlib.pyplot as plt
import seaborn as sns

# inputs

token = ‘YOUR_API_TOKEN_HERE’
endpoint = ‘https://api.realto.io/elia-sf-BE/GetChartDataForZone’
sourceId = 1 # 1 = Belgium
dateFrom = ‘2021-03-07’
dateTo = ‘2021-03-15’

A note on some of the packages here, we’re going to be using “requests” to actually call the API. “re” will be used for running a RegEx query which is needed for our post-processing. Pandas will be used to create the data structure and finally seaborn will be used to create the timeseries plot at the end.

Next, let’s query the API, convert and set-up our dataframe:

# call API and set-up DataFrame

response = requests.get(endpoint, headers={‘OCP-Apim-Subscription-Key’: token}, params={‘dateFrom’:dateFrom, ‘dateTo’:dateTo, ‘sourceId’:sourceId})
json = response.json()
data = json[‘SolarForecastingChartDataForZoneItems’]
df = pd.DataFrame(data)

Here we’re calling the API and then taking the “SolarForecastingChartDataForZoneItems” array and converting it into a Pandas DataFrame. Let’s take a look at what that produces:

One last thing we need to do is append a proper timestamp to each row. You can see that a Unix epoch DateTime currently lives in the StartsOn column in a JSON syntax. Let’s go through each row, parse the DateTime string and pull the numbers out (using RegEx), convert this into a Pandas timestamp, and finally append the new column to the DataFrame.

# create timestamps

dates = []

for index, row in df.iterrows():
dates.append(pd.to_datetime(re.findall(“\d+”, row[‘StartsOn’][‘DateTime’])[0], unit=’ms’))

df[‘DateTime’] = dates

Once that’s done the data is ready to be visualised!

Plotting the Graph:

There’s a couple of Python packages that exist for visualising data- the one we’re using is a package called seaborn. Check out some of the visuals in their examples gallery for inspiration. Since our DataFrame is set-up, getting a timeseries graph is simply a case of defining what columns we want to plot + any settings we want to play with (e.g. colours, titles, sizes, etc). Let’s keep it simple:

# visualising

new_df = df[[‘WeekAheadForecast’, ‘DayAheadForecast’, ‘MostRecentForecast’, ‘RealTime’, ‘DateTime’]]
new_df.set_index(‘DateTime’, inplace=True)
sns.set_style(“darkgrid”)
fig, ax = plt.subplots(figsize=(10, 10))
sns.lineplot(data=new_df, ax=ax, palette=”tab10″, linewidth=2.5)

This outputs the following graph:

You can see the forecasts become more accurate the closer you are to the forecast date… Who would have guessed?

And voila, that’s it!

In Summary:

We queried the solar forecast API, post-processed the data, and visualising the output on a time series graph.

We hope this tutorial proved useful to you. Feel free to reach out to us with any questions you may have.

The full Python script

You can find a copy of our script on our GitHub page.