Airlines are constantly aiming for defining and executing practical big data use cases. And sometimes you get dizzy by all these fancy ideas airlines come up with. Believe it, I am no stranger to that. However, personally I’m always aiming for feasible and executable solutions.
Big data, in my opinion, has definitely survived the buzzword hype and we can witness a multitude of practical use cases, tools, and benefits this technology is accomplishing.
In Airline Operations The Availability Of Big Data Is A Real Problem
However, when talking to airline operations experts we often encounter an interesting problem. The challenge here is that big data or sources for big data are simply not available.
To avoid misunderstandings: I’m not talking about the huge amount of data every aircraft is generating during a flight. I’m talking about process data: data from ground handlers, passengers, timestamps, information about process changes, etc.
When analyzing available operational data from airlines, one usually has to concentrate on data sets with a very limited amount of attributes. Scheduled times, actual times, number of passengers — actually, the basic data.
And the problem gets bigger since every flight is operated not more than once per day. Accordingly, we have to rely on a maximum of 365 sets of data with a limited set of attributes.
Most Big Data Approaches Don’t Work With Limited Airline Data
Ultimately, this results in a situation that most of the big data approaches, all the fancy tools, and analytical approaches simply don’t work. The available limited sets of data actually don’t allay the big data hunger.
What’s the result? Airline operations is still bound to old-fashioned approaches in terms of data analysis and technological possibilities. And you all know these approaches: old-school database, Excel exports, and so on.
But Wait, I Have An Idea To Create Big Data!
As the title of the blog post already relinquishes, I’m absolutely fascinated about a new possibility on the horizon which dramatically changes this situation.
A possibility that literally helps airlines to generate big data in the area of airline operations. Moreover, it creates a whole new source for analytics. No worries, you won’t have to connect a multitude of additional data sources or a vast number of suppliers.
I can see the questions mark above your head — “What is he talking about? How does that work?”
Events To Exponentially Grow Available Data
The core idea is to create rule-based events out of basic data. That means that very basic incoming data, such as off-block events, estimated time of arrivals, ACARS messages, information from ground handler, etc. are processed according to defined rules. And by doing so, you can create dozens of events out of one single data attribute.
Here Are Two Rule Examples
Here’s a basic one:
- An incoming ETA is compared to the STA. In case the ETA is five minutes later than the STA a “delay event” is generated.
Sure, it’s quite boring. So let’s spice it up
- An incoming ETA is compared to the STA. In case the ETA is five minutes later and first-class passengers are on board a “critical delay” event is generated.
I’m pretty sure you can already think about ten, twenty, or hundreds of other rules. Simple and complex ones.
Of course, that requires sitting down and defining the rules that are important to your airline. However, by doing so you can create a huge amount of data for every single flight.
Ultimately, this provides the possibility for a much more detailed and sophisticated analysis.
Rules Have No Limitations When It Comes To Airline Big Data
There are basically no limitations when it comes to rules. You can include data from public sources such as the weather. Or you go for maintenance information, NOTAMS, aircraft data, etc.
You can even take this one step further and create events out of events. Again, with defined rules. For example, an event that is generated when three critical delay events are generated within a timeframe of five minutes.
Our Experience With This Big Data Approach At Airlines
We went down that road in 2019 with a large European network carrier. Therefore, we developed a big data component that allows creating the above-mentioned events.
Afterward, we set up 30 rules for an initial test run. That means each flight of the 2,000 daily flights was processed by our component and respective rules.
The Results Literally Blew Us Away
Although we started with this very limited set of rules, the amount of data increase by factor 30. That means 30 times more data to analyze. 30 times more data to detect problems. And 30 times more data to identify the root causes of problems.
And to give you a number: On average 30 rules (automatically) created 85,000 – 120,000 additional data sets — DAILY! Just imagine what’s possible with 500 or even 1,000 rules.
What Do Airlines Need to Realize That Big Data Approach?
What’s needed for it? Actually, not very much — at least compared to other big data solutions. Yes, you need a software component that generates the events and provides you the possibility to define all the rules you need.
For the techies: We realized that with common big data technologies, by developing a component which is based on Kafka and Flink. Developed with Scala.
And yes, it is an effort to tackle. However, the potential of this approach is so unbelievably huge. Therefore, the efforts are absolutely reasonable.
Personally, I really do think that this approach holds the potential to massively improve airline operations by providing new possibilities for analyzing — and for applying state-of-the-art technologies.