Rearden Commerce deems a decade of Big Data worth crunching to enhance flight search

Rearden Commerce has crunched a decade’s worth of Big Data, namely data on 67 million flights from 24 US airlines over 10 years, in a bid to enhance flight search in its Deem Travel booking platform.

A project of Deem Labs, Rearden personnel used Apache Hadoop, Apache Pig and MySQL to transform and analyze US Bureau of Transportation Statistics data on the 67 million flights from 24 US airlines from January 1, 2001, to December 31, 2010.

While static infographics for blogs have been a data and marketing craze in the travel industry of late, Rearden has gone a step further and created what it calls an “animated infographic” in the form of a YouTube video, Heat Map, Seasonal On-Time Flight Performance for US Airports.

The video, which is “best viewed at 720p resolution,” depicts a color-coded analysis of flight performance on a daily basis over an average year during the decade studied, Rearden says.

Rearden’s goal in all of this “is to overlay the information onto our Deem Travel booking platform, helping users select the best travel routes for any date or context, based on history,” a spokeswoman says.

The company already has integrated a limited amount of this flight reliability data into Deem — with more to come — for its corporate clients and their employees.

“Fliers will no longer have to rely on superstitions when planning travel,” the spokesperson says. “Through Deem, they’ll have historical data to gauge the best route for any domestic flight, any time of the year.”

That is the idea, anyway.

“Right now, when a traveler sees flight search results on Deem, they see historical on-time performance for each segment — as a probability percentage for that flight arriving on-time,” says Steve Bernstein, Rearden’s head of analytics. “This new analysis gives us the ability to guide travelers toward alternate routes. For example, when one segment within a journey has poor historical performance, we can automatically suggest a new connection with better performance.”

Among the myriad findings gleaned in the Big Data analysis of US flights were:

  • There was a 13% chance of a flight departing or arriving late or being diverted or cancelled;
  • If you flew during the 10-year period on December 23, it was the worst day of the year to fly, with a 25% chance of the flight being late, cancelled or diverted;
  • October 3, on average, was the best day to fly with 7 in 1,000 arrivals and departures diverted, cancelled or late;
  • The airport with the worst arrival performance was Dutch Harbor, Alaska, with about a 33% of the flights arriving late;
  • Newark Airport, with 25% of flights arriving late, was the worst performing major in the US in terms of arrivals over the 10-year period;
  • The summer vacation period is just as bad in terms of on-time flight performance as is the winter; and
  • July 4 averages “exceptional on-time performance,” Rearden says.

That is just a taste of what Rearden found — after all the Big Data was huge.

Share on FacebookTweet about this on TwitterShare on LinkedInEmail to someone
Dennis Schaal

About the Writer :: Dennis Schaal

Dennis Schaal was North American editor for Tnooz.



Your email address will not be published. Required fields are marked *

  1. What is Big Data ? – another sterile BI debate ? | Antivia Blog

    […] vendor. A good example of this is the recent press release from Reardon Commerce, which talks about analysis of punctuality data from 67M US airline flights as “Big […]

  2. Eric

    This is very informative. I would like to see it in a more robust approach to consumers seasonal flyers. And this would not just be limited to and from but I also want to see the other factors that justify whys and what ifs.
    just pulling put some points on the article..

    “October 3, on average, was the best day to fly with 7 in 1,000 arrivals and departures diverted, cancelled or late;” – so why October 3?, what was the weather condition? what was ?points that made it unique?

    “The airport with the worst arrival performance was Dutch Harbor, Alaska, with about a 33% of the flights arriving late” – Alaska I can argue that these is bec of the winter season (and went there on October ’05), but how will your smart apps help consumers suggest the best times if I still want to go to Alaska. and also getting the best deals?

    • Steve Bernstein

      These findings were based on patterns that emerged from an analysis of ten years of flight data. The patterns are therefore seasonal and not based on the weather on any specific day. Weather is only one factor that influences on-time performance, though an important one. As important is overall system traffic, with October generally being a light travel time of the year. April is also a slow travel month in general. And in both cases extreme weather is unlikely, so the combination of light traffic and generally moderate weather makes them good times to fly.

      Regarding Alaska, yes, more likely to see extreme weather there in general, and yes, our App helps you choose by price if you prefer, or by many other characteristics, like convenience, flight amenities, etc. Ultimately we’ll also help you consider your chances of arriving on time–we do some of that now, we’ll do more in the future. if you have to fly to Alaska at certain bad weather times of the year, we’ll still try to show what to do to maximize your chances, but even your best choices may not be very good. 🙂

  3. Steve Bernstein

    Great idea Ian, by the way, we’re hiring. 🙂 We do license FlightStats data. Our objective in this instance is to assist the traveler by recommending the best connecting hub given a travel date and hour in the (possibly distant) future. For this specific objective, the licensable FlightStats data and the publicly-available Bureau of Transportation data are identical. While neither serves this objective directly, companies with strong, creative analytical talent on board can use these data sources to innovate ever-better traveling experiences.

  4. Ian

    Why not just license’s excellent on-time performance data feed instead:

    And then go one step further– start factoring in the quality of service within the airplane.


Newsletter Subscription

Please subscribe now to Tnooz’s FREE daily newsletter.

This lively package of news and information from Tnooz’s web site provides a convenient digest of what’s happening in technology that drives the global travel, tourism and hospitality market.

  • Cancel