Natural gas is one of the most important sources of energy, both for Denmark and for the world. Globally, natural gas accounts for 22% of the world’s energy consumption (Shaikh et al. 2017). In 2017, approximately 17% of Denmark’s energy consumption was provided by natural gas (Danish Energy Agency 2018a), and the energy consumption in Denmark based on source can be seen in Fig. 1. Natural gas is regarded as the cleanest fossil fuel and with the world’s policies prioritizing green energy, and environmental impact, natural gas is seen as a good transition to completely clean energy from fossil fuels (Karadede et al. 2017; Zhang and Liu 2017; Liu et al. 2018). One important example of this is China, which has the highest greenhouse emissions in the world due to high carbon energy consumption (Shaikh et al. 2017), where, currently, there is a reduction in coal usage along with the increase in the use of alternative sources, such as natural gas (Liu et al. 2018). Annually, approximately 2.5 billion Nm³ natural gas is consumed by Danish customers, whereas approximately 0.8 billion Nm³ natural gas is exported to Sweden through Denmark (Bruun 2017).

Gross energy consumption in Denmark based on data source originated from Danish Energy Agency (2018a)

This project was facilitated by Energinet, in order to describe the current knowledge in a more structured manner and try to eventually improve their current natural gas demand forecast accuracy, with the aim of providing more accurate inputs for their natural gas balancing market. All the data used are based on historic data provided by Energinet, and Energinet Energy Data Service.

Denmark’s government has the goal to be independent of fossil fuels by 2050 by introducing more renewable energy (Denmark.dk 2018). This is not completely contradicted by the current usage of natural gas, since the same infrastructure can be used for bio-natural gas (BNG), which is similar to natural gas in composition, obtained though from renewable and biological sources. In 2017, the supplied biogas accounted for 5% of the total consumption and more biogas plants are currently being built (Energinet 2017b). Biogas plants usually use manure, agricultural waste or household waste (Sotiropoulos et al. 2018) as input to create raw biogas in a circular manner. Natural gas is characterized by high methane content and biogas can be made into a similar composition by upgrading raw biogas. Raw biogas has a methane concentration of approximately 65%, whereas the upgraded biogas has a methane concentration of around 98% (Energinet 2017b). This means that BNG can be used as a substitute for natural gas, therefore it can replace fossil fuel usage with a renewable energy source. This will contribute in reducing further CO₂ and greenhouse gas emissions in the atmosphere in various ways (Danish Energy Agency 2018b).

Energinet is the transmission system operator (TSO) for natural gas and electricity in Denmark, and is an independent public enterprise owned by the Danish Ministry of Energy, Utilities and Climate. Energinet’s main purposes, as listed on their website, are maintaining a high level of security of supply, integrating renewable energy, and promoting optimal conditions for Denmark’s electricity and gas markets (Energinet 2017c; 2018d). Energinet consists of six subsidiaries and two business service companies, as can be seen in the organizational chart in Fig. 2. The two business service companies are Engineering and Construction and Business Services. Business Services provides IT, procurement, finance, and facility services to the group’s companies in order to support them. Engineering and Construction unit handles the operation and maintenance, and construction works in the Danish gas transmission system (electricity transmission is responsible for the electricity transmission grid). Electricity System Operator is the responsible for electricity markets, system development, and system operation. Gas TSO is responsible for the physical gas transmission system as well as the gas market. Gas Storage Denmark owns the Stenlille and Lille Torup natural gas storage facilities and operates the specific storages. Gas Distribution Denmark is a grid company that distributes gas to 120,000 customers. DataHub provides data services to the participants in the electricity market and optimizes market circumstances for customers (Energinet 2018c, e, f).

Energinet organizational chart based on (Energinet 2018e, f)

Due to the high capacity and continuity requirements, natural gas is usually transported through pipelines from the extracted or stored locations to the final users. The gas pipeline can be divided into 2 sections, the transmission network and the distribution networks. The transmission network consists of larger diameter and higher pressure pipes, aimed at transporting the gas over longer distances towards different distribution lines. The distribution lines consist of smaller diameter and lower pressure lines, stemming from transmission lines and ending at the end users. As the distribution lines need to reach the final consumers, they are more spread out in a network structure.

Figure 3 shows the gas transmission infrastructure in Denmark. The natural gas infrastructure of Denmark consists of a transmission line of approximately 926 km of high pressure pipeline (Energinet 2017a), which is a relatively short and simple system compared to most European countries. The pressure range for the natural gas inside the Danish transmission lines is around 44–80 bar, with 44 bar being the minimum acceptable value. The high pressure is useful for mainly two reasons, firstly the gas flows from high pressure points to the lower pressure points in the distribution network, so the pressure provides the flow of gas. Secondly, high pressure allows for building of “linepack”, which is the stored gas in the pipelines that act as a buffer for fluctuations.

Danish transmission gas infrastructure. Egtved compressor station is shown as a circle. The orange line shows the pipeline, whereas the orange dots represent measurement and regulation stations. Image edited from Energinet (2018b)

During the transport of the gas through the pipeline pressure losses are encountered due to high flow, low diameter, long travel distance, and high internal pipe roughness (Energinet 2018b). Gas flows at around 25 km/h in high pressure pipelines (Chen et al. 2018), therefore the TSO should plan in advance the required amount of linepack or pressure in the pipelines as natural gas can only be supplied to the system at predetermined points, such as storage areas, from other markets or from other subnets via the compressor station. In order to predict customers’ possible usage, short-term forecasting of the consumption is required by the TSO.

As providing security of supply is one of Energinet’s main purposes, many simulations are run in order to be aware and ready for difficult situations, such as extreme weather conditions, maintenance, unplanned simultaneous breakdowns at different facilities and so on. Maintaining security of supply should be balanced with the available capacity and operational costs. Increasing the security of supply might mean that there could be less capacity for flow, on the other hand working with high pressures or increasing the capacity might result in high operational costs (Energinet 2018b). In order to provide security of supply, the linepack must be flexible, meaning there should always be some margin in the pipelines for gas injection, as well as for gas withdrawal. The optimal balance in all these areas should be found in order not to have supply interruptions or to avoid the excessive increase in customers' bills, as significant capital investments are needed to improve the physical gas infrastructure (Ronan et al. 2017).

Linepack consists of three sections: Linepack to support minimum pressure, linepack to support flow, and flexible linepack, which can be seen in Fig. 4. Linepack to support minimum pressure is approximately 40–44 bar, since this is the minimum acceptable pressure in the transmission pipelines in Denmark. Linepack to support flow is required as the pressure in the pipe tends to decrease along the pipeline. The pressure is at its highest pressure levels at the inlet or pressurization location, and decreases with the flow as the distance from the input point increases. Therefore, in order to have minimum 44 bar pressure throughout the pipe, the inlet points must have a higher pressure so there is flow inside the pipe. The first two sections of the linepack are mandatory to keep in order to provide a functioning system. The third section is the flexible linepack, which provides security to keep the system within the acceptable operating limits. The flexible linepack is useful, because, if for example, there is excessive consumption, the flexible linepack will decrease while linepack to support flow will still be maintained. In the reverse case, where the consumption is low and the production is too much, the flexibility allows the system to be kept under the maximum operating pressure. It is, therefore, critical for the TSO to regulate the flexible linepack in order to provide a functioning system without down time. Consequently, good forecasting accuracy is crucial to regulate the flexible linepack in a satisfactory manner.

Balance between market, security of supply and costs. Image taken from Energinet (2018b)

In Denmark, before 2013, the transmission pipelines were all connected directly without separation, meaning that any gas injection or removal would affect the whole system. On October 1st, 2013, with the introduction of the Egtved compressor station, the transmission system became divided into four subsystems. The flow between the four subsystems can be controlled with compressors by bringing the lines to required pressures. The previous system was similar to a bathtub with various entry and exit points, whereas the current system is like a bathtub with four divided sections, which can be seen in Fig. 5. The four sections, which are called subnets, are named after the relative geographical location of the transmission pipelines with reference to Egtved, and they are Vest, Syd, Nord, and East subnets. The pipe volume of each subsystem is 50.20, 66.96, 30.70, and 140.80 m³ respectively (Energinet 2018b).

Entry and exit points of Danish gas transmission system. The image on the left shows the gas system before 2013, the bathtub model, whereas the right image shows the divided bathtub model with the introduction of the Egtved compressor station. Figure generated by the author, according to the model in Energinet (2018b)

For the consumption data for Nord subset, the yearly seasonality is obvious and easy to spot. It is also seen that the yearly consumption levels are close to each other which can be confirmed in Table 1.

In the summer periods, the lowest consumption drops to barely under 10 million kW h, and in the peak of the winter there are days where consumption exceeds 50 million kW h. The same trend can be seen in the other subnets as well (Tables 2, 3 and 4).

In the East Subnet, Table 2 shows that the general consumption has been quite stable over the years 2014–2017.

The descriptive statistics seen in Table 4 shows a quite stable mean consumption with a high deviation in the minimum consumption.

As expected, natural gas consumption increases in the colder winter periods and decreases when the temperatures rise in the summer. This is a result of natural gas being used for heating purposes when the outside temperatures drop.

The descriptive statistics analysis of the four different subnets is graphically shown in Fig. 6.

Descriptive comparative statistics for Nord, East, Syd, and Vest subnet per year

The distribution pipeline in Denmark is approximately 17,000 km long. Pressure of the main distribution pipeline is 16–40 bar depending on the location. The connection from the transmission lines to distribution lines is provided by 43 measurement and regulation (M/R) stations, which help regulate the pressure according to the requirements of the specific grids. Via the distribution grid, the natural gas is transferred to approximately 400,000 private households and commercial users. As the gas reaches the final consumers, the pressure is lowered depending on the requirements, even down to 20 mbarg (Energinet 2017b).

The research carried out in this paper took place at the city of Herning, Denmark with the years 2018–2019.

2.1 Natural gas consumption forecasting

There are numerous forecasting models available, and many things to consider when choosing the right model in order to extract the maximum information possible from the data at hand. Some of the important parameters can be readily found in the literature review table, such as the forecasting range, forecast application area, available variables, and the amount of data at hand. Moreover, there are observations made from the preliminary analysis, such as the double seasonality of the data which is crucial to be incorporated into the model.

The amount of data at hand is also an important aspect, as in certain situations the data-set might be very small or filled with outliers which makes it hard to fit. In these cases special attention to models that could generalize on these should be selected. In other cases, the forecasting model to be used might require a lot of data to proceed, such as artificial intelligence models. Forecast application area is important, as an industrial company might prioritize and benefit from a different approach than a city. Even in terms of cities, an industrial location and a residential location will exhibit different behaviour so it is important to consider this information while choosing a model.

The literature regarding natural gas consumption forecasting is not as extensive as in the case of electricity forecasting (Chen et al. 2018; Panapakidis and Dagoumas 2017; Panagiotidis et al. 2019). This is mainly due to electricity being a much less flexible commodity (Karabiber and Xydis 2019), meaning that it is not easy to store, and the balance has to be much more precise compared to relatively large pressure range in the line pack in natural gas pipes. Having said that, natural gas forecasting is equally important for shippers, TSO’s, DSO’s, shippers, traders, and players on the market—recently many studies were done reflecting this importance (Karabiber and Xydis 2019).

One of the most comprehensive literature surveys of natural gas consumption forecasting is by Soldo (2012). He classifies the literature on natural gas consumption forecasting by different parameters such as publishing year, forecasting horizon, applied area, data frequency, input data types, and forecasting tools used. His paper covers the literature until the end of 2010. A more recent study, that is published by Tamba et al. (2018), focuses not only on consumption forecasting, but also on production, prices and income elasticity, volatility in the market, and price hikes. In this literature review, which builds upon Soldo (2012)’s work, a similar classification is used, and the papers are covered until the end of 2015. At the end of the paper, Tamba et al. (2018) makes some suggestions for future literature. One suggestion is that the input variable forecasts, such as weather forecasts, should be used instead of their actual measurements, in order to observe the effects of the forecasting variables’ inaccuracies. It is a quite crucial point, as the users of the forecasting tools do not have the actual values of some variables into the future, and the error in the input variables will affect the accuracy of the forecasts. In order to address this issue, it is always important to work via sensitivity analysis, to show the effect of the error in the input variables to the accuracy of the forecasts. Extending Soldo (2012)’s and Tamba et al. (2018)’s work, some of the notable papers on natural gas consumption forecasting from the beginning of 2016 until August of 2018 are examined. The same classification set by Tamba et al. (2018) is followed as it is comprehensive and useful for identification.

In the natural gas forecasting area, some topics require emphasis, as they are quite differentiated from the rest. One such topic is the demand forecasting of China’s yearly consumption. This topic is quite significant and visited by many authors, as China has seen huge growth in the last 30 years. Natural gas consumption has seen even more growth than China’s economic growth and this requires special attention in the planning of policies and resources (Zeng and Li 2016). Similar to China, as a growing country, Turkey has also seen a similar attention on different levels of forecasting.

2.2 The grey model and the generalized Weng model

Zeng and Li (2016) worked on an improvement of the grey model, which they managed to train with a very small data set consisting of nine points. Their model automatically adjusts model parameters, and provides lower MAPE than the other Grey Models tested. They forecast consumption in China for the next 5 years and provide policy suggestions. Liu et al. (2018) made a long-term forecasting model for China, with a novel approach for this problem using a per capita perspective, considering variables such as natural gas price, income, length of pipe, natural gas population, household size, energy substitution, ambient temperature, and central heating. Shaikh et al. (2017) use two different optimized grey models in order to forecast China’s long-term natural gas consumption. The models used accurately capture the nonlinear growth pattern of the forecasted consumption data. Shaikh and Ji (2016) also forecast China’s long-term natural gas consumption using logistic modelling analysis. They present a good overview of their forecast and of the other forecasts for the same period and area. The Diebold-Mariano test is utilized to compare the forecasting results. Their results are also comparable with more complex models, signifying that over-complexity is not required to address this forecasting problem. Zhang and Liu (2017) combine the grey neural network model with the global searching feature of the particle swarm optimization model for the long-term gas load forecasting of a province in China. Although the results are promising, it would be good to have the model tested in multiple provinces. Wu and Shen (2018) use a particle swarm optimization algorithm in order to optimize the coefficients for the proposed grey related least squares support vector machine approach. The results show that the accuracy surpasses the conventional grey model and Artificial Neural Network for yearly forecasting of China’s natural gas consumption. Fan et al. (2018) use a combination approach using grey model and self-adapting intelligent grey model, and having a genetic algorithm to determine the weight of each model. Annual share of natural gas in the total energy consumption is used as a means to determine the level of the data. The results show improvement in forecasting accuracy and also statistical tests are applied to show the statistical significance of the forecasting improvement. Ma and Liu (2017) propose a new model combining the grey model and the polynomial model. The comparison of the combination to other similar models shows an improvement in the forecasting error. Wang et al. (2017) propose a hybrid grey model in order to forecast the yearly natural gas consumption of 30 regions in China. The study shows lower error value compared to other studies, however, the test set used in cross-validation only comprises two points, even though the results are tested in 30 regions. Ding (2018) uses a hybrid method which is a novel, intelligent, univariate grey model to forecast China’s yearly natural demand, and results show smaller error than conventional grey models. The future forecasts are similar to international agencies and other researchers, justifying the results.

Li et al. (2016) forecast China’s long-term natural gas production using the generalized Weng model. The main data used is the estimation of the reserves remaining in China. The main weakness of the study is that no error metrics are utilized in order to measure the accuracy of the forecast. Instead, the forecasted values are compared to other estimates such as IEA’s World Energy Outlook 2015 and BP’s Energy Outlook 2035. Zhang and Zhou (2018) Use a combination of Boltzmann model and a polynomial curve to forecast China’s yearly natural gas consumption. The combination offers lower error than both of its components and greatly increases forecasting accuracy. Wang and Lin (2017) estimate the consumption peak for China’s natural gas consumption. The paper includes a detailed factor and sensitivity analysis, and includes long term consumption forecasting, however, similar to Li et al. (2016), no error metrics are provided. Chai et al. (2018) use decomposition tools and statistical tests to choose variables which show that urbanization carries a significant effect on future natural gas consumption. Partial least square regression is used to build the forecasting model to predict China’s yearly natural gas consumption and it yields low forecasting error. Results are presented as scenario analysis.

2.3 Long-term natural gas forecasting

Karadede et al. (2017) introduce and test a breeder algorithm approach for coefficient optimization for a nonlinear regression for long-term natural gas forecasting. Similar to Karadede et al. (2017), Wang et al. (2018) use a hybrid algorithm where the particle swarm optimization method is used to optimize the parameters of the wavelet neural network. This helps with a global search that avoids local minima in neural network optimization. The model is used for forecasting three scenarios for the long-term natural gas consumption in China. Ervural et al. (2016) use a hybrid algorithm where genetic algorithms are used to choose parameters to forecast monthly consumption of Istanbul, which is a city with over 15 million inhabitants. The results of the study are not presented in depth although it is seen that the genetic algorithm autoregressive moving average (ARMA) hybrid method performs better than the classical ARMA model. Ozdemir et al. (2016) use a hybrid approach where a genetic algorithm-simulated annealing method is used to estimate linear regression parameters for long-term forecasting of Turkey. The results are compared with a previous paper and seem to achieve better accuracy. The forecasts are used to construct two scenarios for future consumption.

Akpınar and Yumusak (2016) forecast the year-ahead natural gas consumption with monthly frequency using univariate statistical methods for a city. The methods tested are time series decomposition, Holt-Winters exponential smoothing, and Autoregressive Integrated Moving Average (ARIMA) variations. The models with the highest accuracy are ARIMA and Seasonal ARIMA, and it is presented that as the complexity of the model increases the accuracy increases for the tested models. Chen et al. (2018)’s work revolves around comparing different autoregressive forecasting models. In general, the exogenous models outperform models without exogenous variables. It is mentioned that when nomination data is included, temperature input becomes insignificant, however, this is most likely due to the nominations are being done using temperature predictions. Proposed FARX model outperforms other AR models tested, nevertheless other models that utilize exogenous variables, such as ANN or ARIMAX models are not considered in the comparisons.

Panapakidis and Dagoumas (2017) present a good overview of artificial intelligence based models in their literature survey. They work with datasets with many missing values, therefore they use wavelet transformation in order to smooth out the data. Then, a novel hybrid algorithm that consists of ANFIS-GA-FFNN is applied. Each region is inspected separately. The test data set is 6 months so the data is not tested in the second half of the year. The data is tested with various regions and the results are presented clearly with various error calculation methods. Akpınar et al. (2017) worked on a hybrid method in order to make short term forecast for regional level. They use artificial bee colony trained ANN and show that it has better accuracy than backpropagation trained ANN. Merkel et al. (2018) compare the forecasting results of a deep neural network (DNN) to linear regression and a traditional ANN, and points out that the DNN outperforms both in various occasions. Some other important points mentioned are that the deeper the network, the better results are achieved, although DNN does not outperform in every region so it is suggested that various forecasting methods should be evaluated together. Bai and Li (2016) forecast the daily natural gas consumption for a city in China. It is shown that SC-SVR model outperforms backpropagation NN and least squares SVR methods for the given dataset. A weakness of this study is that there is a selection bias in the test set as only the last 60 days of the year are tested. Therefore, the accuracy metrics are not fully reliable as it does not include a full year nor exceptions such as the Chinese holidays.

Khani and Farag (2018) propose a new two-stage model where the temperature dependent and temperature independent parts of the gas demand are evaluated in parallel, and the online data is used for calibration. The forecasts can be done for hourly and intra-hourly intervals. The results are compared with ANN and regression tree models and the proposed approach show lower error and processing times. The real world challenges of the proposed model are also discussed. Naim et al. (2018) show that the TBATS model performs well with data with multiple seasonality and it is also seen that using MAPE errors is better for training than using RMSE errors for the TBATS and BATS models. Kovacic and Dolenc (2016) forecast the monthly natural gas consumption for a chemical processing plant in Slovenia. The chemical plant accounts for 1.47% of the natural gas consumption of Slovenia. Genetic programming and linear regression are applied in order to minimize a cost function. The most successful model constructed by the genetic program is applied in real life and is seen to be superior to expert predictions in minimizing the cost function.

Baldacci et al. (2016) use two forecasting models, one based on a similar day and one based on local regression approach. Local regression approach seems to spot anomalies with high precision. One significant contribution of the paper is the in depth investigation of consumers’ behaviours to turn on heating based on temperature changes. Another important point of the paper is the focus on anomaly prediction which could be a useful tool for transmission and distribution system operators. MARS, CMARS methods proposed by Özmen et al. (2018), which are kind of piecewise polynomial regression models, outperforms NN and Linear Regression and returns high accuracy and lower variance. In the study, the test period is kept at one year which shows that the model performs well over a relatively long time frame. Gascon and Sanchez-Ubeda (2017) also investigated the piecewise linear additive models, and proposed the SNAKE method. Their suggested method is seen to be competitive against neural network models. In addition, the base temperature for heating degree days (HDD) is not set in advance and is determined by the model.

Duangnate and Mjelde (2017) investigate how the variable selection and number of variables effect the forecasting accuracy. Despite their conclusion being ambiguous, they suggest that various methods should be tested, and the best method should be identified according to available data and the forecasting need. They also show that the factor models have beneficial information for forecasting. However, too many parameters negatively affect forecasting performance. They have also shared their code and raw data which enables repeatability of their work. In Scarpa and Bianco (2017)’s meta study for the quality of natural gas consumption, they investigate the accuracy of long-term yearly forecasting models using standard regression and argue that the input variable accuracy, such as gross domestic product (GDP), heating degree days (HDD), and the price is crucial to forecasting accuracy. Akouemo and Povinelli (2016) work on anomaly detection for natural gas demand. Their proposed method significantly increases the accuracy of linear regression by eliminating the outliers in the training set. Hribar et al. (2017) present empirical models and argues that they are valuable to understand the relationship between the input data and the dependent variable. The accuracy of the forecasts could have been better presented. Ronan et al. (2017) successfully implement a composite weather variable (CWV) that takes into account many weather variables in order to forecast peak day consumption and provide an insight into how TSO’s plan for peak consumption events. Their study also backs up Soldo et al. (2014)’s findings in showing that using solar radiation data helps get better forecasting results. In Table 5 there is an analytical list of published papers according to different classifications. Table 5 presents the applied statistical mathematical, economical, artificial intelligence, and hybrid-combinational models on their application areas. It compares the main input data sources and their forecasting horizons.

Except for BNG, natural gas entry to Denmark takes place either from the North Sea or from Germany. From these points, the natural gas passes through the transmission pipes and is either used in Denmark, stored in one of two large storages, Stenlille and Lille Torup, or it is transferred to Sweden or Germany. Denmark has a two-way transaction with Germany, meaning the gas can both be imported and exported. On the other hand, Sweden has no natural gas generation other than biogas, which is in small quantities, thus Sweden relies solely on the imports from Denmark. Sweden has no other natural gas source, except some small amounts of BNG production. The physical gas supply chain explaining the flow of the gas from the production to the final consumer is shown in Fig. 7 (Karabiber and Xydis 2020).

Physical gas supply chain

Usually, BNG does not enter the transmission line directly, as it would need a lot of pressurization, instead it is injected into the distribution lines where the pipes operate with lower pressure. This causes some hardship as natural gas consumption is usually measured at the transmission system points, thus a rise in BNG production would mean less gas offtake from the transmission system. Hence, while forecasting the natural gas consumption, BNG production can also be an important parameter, especially in regions where BNG production is extensive.

The natural gas used in Denmark comes mostly from the North Sea, especially from the Tyra offshore platform. Due to the subsidence in the platform, it is planned to be shut down for maintenance from November 2019 and until July 2022. This means that Denmark will be depending heavily on the imports from Germany as less than 10 percent of normal production levels will be supplied from Tyra. This will cause a lower flexibility and, consequently, the need for a better forecasting model arises (Energinet 2018g).

A sample gas infrastructure is shown in Fig. 8. The transmission lines that are owned and run by Energinet carry odourless natural gas at high pressures of about 40–80 bar. Passing through M/R stations they are transferred to distribution pipelines run by DSO’s at a lower pressure. At this point, an odorous substance is added to the gas for safety reasons. Through the distribution pipelines, the pressure is further reduced depending on the needs of the consumers, thus the gas reaches the industrial or household consumer. Additionally, in the distribution lines there are also M/R stations to further regulate pressure, and measure consumption.

Physical gas infrastructure and owners

The transfer of the gas through the physical infrastructure is not analogous with the market’s actions. An interesting example of this is that, despite the physical gas flow from Denmark to Sweden is one way, a power plant in Denmark is allowed to buy BNG from Sweden. This is merely a market transaction, as the physical gas supplied to the power plant is the gas available in the pipeline. Figure 9 shows the interaction between the physical gas transfer and the gas market.

Interaction between the market players and the physical gas transfer

In order to trade natural gas, the shippers first need to reserve the required amount of capacity in the physical pipeline. The capacity reservation can be made yearly, quarterly, monthly, daily, or for the rest of the day. Afterwards, the shipper makes a nomination, informing Energinet how much gas inflow and outflow is expected for the upcoming gas day. The system is built in a way so that the shippers are incentivized to balance their portfolios. Energinet informs the shippers five times a day about their offtake and the shippers can re-nominate or buy and sell gas to balance their portfolio. After the gas day, allocation is done, and the balance of the end positions of the shippers can be seen through allocations made (Energinet 2018a).

There are different platforms for trading gas and buying capacity. The capacity for Ellund and Dragør is sold on PRISMA platform, whereas the capacity for Nybro, Exit zone, and BNG Entry is sold on the Energinet platform. Gas trading in Denmark is made via Gaspoint Nordic system, which is a market open ceaselessly (Energinet 2018a).

As a principle, the transmission system should always be in balance, however due to the linepack, it is possible to have imbalances on the short term. In order to provide balance, Energinet incentivizes shippers to balance their positions, and, in addition, it can also buy and sell gas on the exchange, and inject or withdraw gas from the storage facilities. The nominations and consumption forecasts are used to regulate the flexibility and linepack. Each day the green zone and the yellow zone for the balancing system are calculated using the nominations and consumption forecasts, and published on Energinet Online. The green zone, which is the area in the middle between two black lines in Fig. 10, shows the recommended balance of the pipeline, where Energinet is not allowed to trade. In the yellow zone, which is above or below the green area, Energinet can buy and sell gas in order to balance the system. It is seen that the zero balance in the green line is not symmetrical and they are calculated before each gas day depending on the linepack and Energinet forecasts. Players in the market that are out of balance at the end of the day “cash-out”, meaning that whatever imbalances, which they have are bought by the TSO.

System commercial balance chart graph from Energinet online

In Denmark, there are two main underground natural gas storage sites, Stenlille and Lille Torup. These sites are crucial as they serve two important purposes for the security of supply. Firstly, they can store natural gas that can be used in case of interruptions in the supply by providing a buffer for injection. Secondly, they can also be used to withdraw gas from the system when there is too much pressure, that is, in instances of at high production and low consumption.

Both storage facilities work under their owner, Gas Storage Denmark A/S. Unlike Energinet, Gas Storage Denmark is a for-profit company and they sell storage services on the northwestern European gas market (Energinet 2018f). Energinet owns some of the storage reserve and has the right to inject or withdraw gas from the storage in order to balance the linepack.

Lille Torup storage is located at the Nord subnet where the natural gas consumption is the second highest after the East Subnet. Lille Torup is a cylindrical underground salt reservoir of 5 km height and diameter that consists of 7 salt caverns. The total storage volume available is 4965 GWh which is equivalent to 435 million Nm³. Stenlille storage is located at Zealand region in the East subsystem, which has the highest natural gas consumption of the four subsystems. It is an aquifier type storage facility that uses clay-sand as an enclosing “pillow” at 1 km depth from the ground. The total storage volume available is 5855 GWh, which is equivalent to 513 million Nm³ (Gas Storage Denmark 2018; Energinet 2018b).

Focusing on each of the points, forecasting range along with forecasting frequency is one of the most important aspects of the data. It determines if the seasonality patterns emerge or not. As an example, considering a data that has daily seasonality with hourly frequency, and 2 day (48 h) forecasting range is required. In this setup one has to consider the daily seasonality in the forecasting model. If the data frequency was daily, it would not be required, as with the aggregation of the data, the seasonal pattern would disappear. On the other hand, if there was yearly seasonality, depending on the model and the strength of seasonality, it could possibly be omitted in the 48 h forecast as the yearly pattern would not emerge in such a pattern. The frequency of the data is also important as it is closely connected to the number of samples. Again, an hourly data would have 168 data points for a week, whereas a daily data would only have 7 points for the week.

A gas day is defined as the time period in which the natural gas consumption is measured starting from 06:00 am until the next day 06:00 am. The date of the gas day is day includes its starting time, for example, the gas day for 5 May 2017 starts at 5 May 2017 at 06:00 am and ends at 6 May 2017 06:00 am. All the dates mentioned in the study will refer to gas day for Denmark’s time zone, which is in UTC + 1.

A forecasting problem is often approached without knowing what is the correct algorithm or a model to be used. As long as the exogenous variables carry information about the dependent variable and they are correctly chosen and used; the models that use the exogenous variables are supposed to have better accuracy compared to the models that do not utilize those (Duangnate and Mjelde 2017). However, it is also equally important not to overcomplicate the model with unnecessary or redundant variables. This creates the risk of overfitting, therefore lowers the accuracy of the forecasts. As such, each variable taken into model must have a good justification and must be eliminated if necessary in order to simplify the model and avoid overfitting.

Energinet carries out many different forecasts, ranging from very short range to decades ahead. One of their forecasting tools, of which this study aims to increase the accuracy by implementing various models, is their day-ahead forecaster. This forecaster is adopted from an electricity forecasting model, therefore it is assumed that there is possibly room for improvement and specialization for the task.

Gathering all the information on the system balance, the four different subnets structure, the gas supply chain, the gas infrastructure, as well as the interaction among all different market players and the physical gas transfer, the outcome was to map the system needs in order to move on to more accurate forecasting. The current practice was to have the same electricity-oriented forecaster for the natural gas market. There are a number of parameters that are not sufficiently researched or even at times omitted via the current prognostic tool. Main uses of natural gas are residential heating, industrial uses, public transport, and power generation, with residential heating taking the lead. While using the temperature variable in the forecasting models, some other forecasters often use Heating Degree Days (HDD as a conversion from temperature. The reason is that natural gas is used mostly for heating, and heating requirements arise below some base temperature. The heating requirement is proportional to the temperature below the base temperature and HDD provides a good metric combining the time and “coldness” in one value.

Another parameter is the day type, e.g. if it is a weekend or a weekday. Although sometimes, the single weekdays that are standing in between two non-weekdays are also marked as holidays as people often tend to take those days off and they show similar consumption characteristics to holidays.

Another one is, of course, the highly associated meteorological conditions, such as wind speed, wind chill, rain amounts, solar radiation, maximum & minimum temperatures etc. The variability of all different meteorological data has a strong influence to the whole system, equal to the energy demand and prices, aiming to optimize each model’s parameters, in order to obtain the lowest possible error value. It should be stated here, that it is expected that the error variation will increase towards the end of the year, where it becomes quite hard to forecast due to cold weather and the holiday season.

The selection criteria for choosing each time the appropriate forecasting model for each application are determined based on: (1) the option for supporting double seasonality, (2) supporting exogenous variables, (3) suitable for day ahead prognosis, (4) ease of use. Some of the most known forecasting models include (1) ANN Models (nnetar—forecast package (Hyndman 2018a)); mlp– nnfor package (Kourentzes 2017b); elm—nnfor package (Kourentzes 2017a); (2) ARIMA Models (auto.arima—forecast package (Hyndman 2018b)); 3) Benchmark Model (tbats—forecast package (Hyndman 2018c)).

Along with the identified main models, benchmark models are also chosen and forecasts from the TBATS will be used to provide a benchmark. TBATS is a simpler model without exogenous variables.

5.1 ANN model

ANN models have been widely used in the literature and seen to provide satisfactory forecasting accuracy. ANN’s are a machine learning system that resembles the neural networks of the biological structures (Krogh 2008; Li and Shi 2010). One important aspect of the ANN’s is that, by having a hidden layer, one can achieve a nonlinear output function. In a feedforward neural network, the information is carried forward from one layer to the next layers and not backwards. The weights of the connections are updated in each run, which is called the training of the network. A DNN represents an ANN with 2 or more hidden layers. A fully connected neural network refers to an ANN that each node is connected to all the nodes in the next layer with a specific weight (Hyndman and Athanasopoulos 2018).

The structure of a sample ANN with 3 layers can be seen in Fig. 11, with 4 nodes in the input layer, 6 nodes in the hidden layer and 1 node in the output layer. Each ANN is comprised of an input layer, an output layer, and nodes in the associated layers. Optionally, single or multiple hidden layers can also be present. Each node sums its inputs multiplied by the respective weight and the summation is transferred to an activation function with a bias. A common activation function is the sigmoid function which is represented as seen in Eq. (1).

A Fully connected Multilayer feedforward neural network with 3 layers and a sample ANN node

The result of the activation function is the output of the node. A sample node and calculation can be seen in Fig. 11. During training of the conventional ANN, the weights of the inputs and bias are updated in order to minimize the error. One common algorithm to optimize the weights is called the backpropagation algorithm. In this case of forecasting, the number of input nodes is equal to the number of variables that are inputted into the system. The output node is always one, which gives the point forecast result. The number of hidden layers and the nodes in each layer will depend on the model and heuristics testing. An ANN without a hidden layer will be analogous to a linear regression model. Adding more layers and nodes will enable the model to perform non-linear and more complex tasks, whereas adding too much of them may cause overfitting and cause the model to lose its generalization ability. In the case of overfitting, the noise in the data is mistakenly interpreted as information and is contained in the data (Hyndman and Athanasopoulos 2018).

5.1.1 NNETAR

NNETAR function in the forecast package in R is a feed forward neural network that contains one hidden layer. It uses the lagged variables as the input to the model, therefore it fits a neural network autoregression (NNAR). NNAR(x, y) model uses x number of lagged variables and has y number of hidden nodes in its hidden layer in a non-seasonal application. The number of lags are determined by testing the models according to Akaike’s information criterion. AIC estimator penalizes the number of variables while valuing the quality of the fit, therefore it allows for selection of a good model with a minimum number of inputs. In case the data has external regressors, they are also input into the system similar to the lagged variables. Because the neural networks have random weights as starting points, they are better trained with a large number of data. In the nnetar model, the network training is repeated for a set number of times and the mean of the results are outputted (Hyndman 2018a; Hyndman and Athanasopoulos 2018).

5.1.2 MLP

MLP is a function in the nnfor package in R. It sets up a multi-layer perceptron (MLP), which is a fully connected feedforward neural network. The mlp function is therefore similar to nnetar, however, it provides more flexibility than nnetar as it is possible to have more than one hidden layer. In addition, it is different from nnetar because it uses a different neural network package in order to train the ANN. mlp function will be used to set up DNN’s for forecasting in order to check for possible improvements in the forecasting accuracy. DNN’s in some cases may provide more accurate results than single hidden layer ANN’s (Merkel et al. 2018).

5.1.3 ELM

ELM function in the nnfor package in R uses the extreme learning machine (ELM) approach in order to forecast the series. An ELM is a special type of feedforward neural network that does not require iterative training and is set-up much faster than a backpropagation trained ANN. In the ELM, the hidden nodes are randomly generated and do not require to be fully connected. In the single hidden layer feedforward neural network, the weights of the input to hidden layer connections are random, whereas the weights of the hidden layer to the output are determined analytically. The advantages of the ELM is the fast training, it is less likely to converge to a local minima than a gradient-based learning, it provides good generalization performance and the possibility to use a wide range of activation functions (Huang et al. 2006). More details regarding the ELM approach can be found in Huang et al. (2006), Huang (2015).

5.2 ARIMA models

ARIMA models are one of the most frequently used and recognized forecasting models. In order to use a certain data with ARIMA, the data has to be stationary first. This means that the trend and the seasonality of the data should be eliminated before applying the ARIMA model. ARIMA model consists of 3 parts, autoregressive (AR), integral, and moving average (MA). AR refers to using the lagged inputs to forecast the future data. Integral refers to differencing needed to make the data stationary. MA is similar to AR, as it is used to forecast the error using the past (lagged) errors. (Hyndman and Athanasopoulos 2018). With this brief explanation at hand, ARIMA(p, d, q) model means that the data is differenced d times, p lagged points are used in AR and q lagged errors are used in MA. An ARIMA(p, 0, q) model can also be referred to as ARMA(p, q) model as there is no differencing involved. The regression (Eq. (2)) with ARMA errors (Eq. (3)) with one external regressor can be defined as follows where both equations are determined simultaneously

The result of the activation function is the output of the node. A sample node and calculation can be seen in the Sample ANN node figure (Fig. 11). During the training of the conventional ANN, the weights of the inputs and bias are updated in order to minimize the error. One common algorithm to optimize the weights is called the backpropagation algorithm. In this case of forecasting, the number of input nodes is equal to the number of variables that are inputted into the system. The output node is always one, which gives the point forecast result. The number of hidden layers and the nodes in each layer will depend on the model and heuristics testing. An ANN without a hidden layer will be analogous to a linear regression model. Adding more layers and nodes will enable the model to perform non-linear and more complex tasks, whereas adding too much of them may cause overfitting and cause the model to lose its generalization ability. In the case of overfitting, the noise in the data is mistakenly interpreted as information and is contained in the data (Hyndman and Athanasopoulos 2018).

5.2.1 AUTO.ARIMA

AUTO.ARIMA is a function in forecasting package of R that is used to make an ARIMA fit using the Hyndman-Khandakar algorithm to choose the model parameters (Hyndman and Khandakar 2008). The algorithm first uses the Kwiatkowski-Phillips-Schmidt-Shin (Kwiatkowski et al. 1992) unit root test to identify if the series is stationary and to choose the correct d parameter. Then various p and q parameters are heuristically tested and selected in order to minimize the corrected AIC. The function is not fully automated, and the outliers have to be cleaned, and if there is variance Box-Cox transformation should be applied before using auto.arima. The fitting is done using MLE. After the fit the results should be examined for irregularities such as autocorrelation. When using the auto.arima function with exogenous variables, the function works as a regression model with ARMA errors. A detailed explanation of the procedure and best practice can be found in (Hyndman 2018b; Hyndman and Athanasopoulos 2018).

5.3 Benchmark models

Benchmark models are simple models that are used as a comparison to see if the more advanced models are performing well. A common benchmark model is the naïve forecast which assumes the data to be forecasted is equal to the previous value. In seasonal data, seasonal naïve forecast is used and it assumes the point to be forecasted is equal to the previous seasonal value. For example, in a weekly seasonal data, the next Monday’s value will be forecasted as being equal to the previous Monday. Using the MASE criteria will enable a comparison between the naïve forecast giving a natural benchmark. In this study TBATS is selected as a benchmark for the main forecasting models. The other benchmark is the current forecaster that Energinet uses.

5.3.1 TBATS

TBATS function in R is used to set-up the TBATS model. The main advantage of the TBATS is that it allows for modelling multiple seasonalities and it works well with frequent data, such as daily or hourly. It even allows for modelling dynamic seasonality. It is also possible to model the errors as an ARMA model. The function uses AIC criteria to automatically select if Box-Cox transformation is needed, if the data has a trend or damped trend, and to use the ARMA errors or not. It is a powerful method that decomposes the data into a trend, and seasonalities and uses Fourier series to fit the seasonalities (Karabiber and Xydis 2019; De Livera et al. 2011; Hyndman 2014, 2018c).

In order to improve forecasting accuracy and maximize the extraction of information incorporated in the exogenous variables, future work could focus on optimizing and improving the existing variables versus Energinet’s current model. Having all recent literature and existing natural gas forecasting models in place, suggestions for improving the forecasting methodology could follow the basic steps suggested by Hyndman and Athanasopoulos (2018) with some specific customizations for the forecasting task at hand. An extended methodology can be applied to any forecasting problem with exogenous variables. At a later stage an improved forecasting model could be creating an optimized single weather variable, and making a consumers behaviour analyses and more effectively taking into account also heating/cooling seasons, and delayed consumer response. This could yield a better understanding of the effects of special days and holidays and could help reveal the exact relation between the weather effects and consumer behaviour.

This study was focused on collecting all the latest articles on natural gas forecasting in relation to the Danish market and laying the foundations for future work on improving forecasting of the Day-Ahead Natural Gas Consumption in Denmark. The interaction among the market players and the physical gas transfer was illustrated based on the physical gas infrastructure and the gas market, that was till now vaguely described. The physical gas supply chain was displayed in relation to the end user and the gas market. Based on this work, researchers can have a starting point against a number of options presented, such as ANN and ARIMA models. The goal has always been to improve the forecasting accuracy versus the benchmark model of TBATS in a so much different market environment compared to the electricity one. Having the literature and the most used natural gas prognostic models at place (such as NNETAR, ELM, MLP ANN models, and AUTO.ARIMA function of ARIMA models, compared against the tbats benchmark model) suggestions for improving the prognostic methodology could follow the comprehensive work suggested by Hyndman and Athanasopoulos (2018), which presents enough information about each method, so the users to be able to move forward with the selected forecasting in a sensible way with some specific customizations for the forecasting task in hand. This way, an extended methodology can be applied to any forecasting problem with exogenous variables and complicated seasonality patterns.