Big Data: Bringing the magic back to travel technology

Last week I was invited to speak on a panel at the Boston THack, Tnooz’s event for developers and engineers. The topic was innovation in travel technology.

At some point during the discussion at ITA Software‘s offices, Glenn McDonald, a user interface engineer at Google, who was sitting next to me, commented that “innovation isn’t really a tangible thing”.

It’s more a reflection of how we perceive something tangible. Like magic. When we say something is magical, that’s another way of stating that we don’t understand how it works, McDonald surmised.

Real innovation isn’t that much different from magic; it occurs when someone solves a problem by removing a constraint that everyone else had taken for granted.

McDonald’s analogy inspired me to make two far less elegant and much more incendiary comments, such as “APIs are evil” and “XML is dead; we just don’t know it yet”.

This, inevitably, has created a bit of a stir, and I can understand why I need to provide some further explanation.

Of course I don’t actually believe that programming interfaces are intrinsically bad, and I’m sure that XML will be around for decades to come.

What I meant is that the ways in which we store, process and exchange data have become obsolete.

Almost everything in online travel today was engineered under this constraint: data costs money, so we need to be as efficient as possible in handling it.

This has defined how our products and companies have evolved. Data has been locked away in silos. Applications talk to the outside world through throttled peepholes that are only capable of answering specific questions with specific answers.

Big Data is the next frontier. Jim Gray called it the Fourth Paradigm, “analogous to when the printing press was invented”.

It has already revolutionised Internet search, social media, astronomy, atmospheric science, genomics, biochemistry, and military surveillance.

In Big Data there are no requests, no predefined parameters and no structured responses. You are free to intersect anything with anything. You can analyse, mutate, group, split, reorder in any way you can imagine.

There are no limits to what you can ask and how often you can ask it. The only constraints are your imagination and the number of computers you have at your disposal.

Suddenly, like magic, anything is possible.

Big Data is not just a set of technologies that can be used to solve hard problems, it is a new way of looking at the problems themselves. And those who master these technologies gain an unfair competitive advantage.

Consider this single example: speed is one of the most important ingredients of a good search experience. In the last fifteen years, no travel company – startup, OTA, airline or GDS – has been able to build and launch an airfare search that returns results in under a second.

One company, a pioneer of Big Data that we all know well, coming from outside of the industry, has just achieved it in less than six months.

We as an industry, and as a community, should consider this our wake-up call. We need to learn the craft of Big Data. We need to invest in the technology of massively parallel processing and scalable storage systems.

Until recently, these technologies were locked away in the IP vaults of big search companies.

But the rise of social media, which has produced more data than had been previously created in all of history of mankind, has forced engineers to re-invent storage, processing, communication protocols – everything.

And Open Source has made these technologies available to any and all of us. Today, they include Hadoop, Disco, Cassandra, HBaseCouchDB, MongoDB and others.

These will evolve over time, but what matters is that learning how to use them will forever change the way we build products and companies.

We also need to take a hard look at all the historical and real-time data that we control, and identify what we can release to the world. We need to create large, open and free data-sets and we need to start publishing real time data streams.

We will be giving up some control and the outcome might appear uncertain. But we cannot not err on the side of excessive caution, because the rewards outweigh the risks. And the cost of doing nothing is greatest of all.

If we do this, others will be inspired to do the same. And this will lead to unpredictable and extraordinary things.

NB: Chime in on Twitter: #bringthemagicback

Share on FacebookTweet about this on TwitterShare on LinkedInEmail to someone
Frederic Lalonde

About the Writer :: Frederic Lalonde

Fred Lalonde is the founder and chief executive of Hopper and also chairman of tnooz.

Previously, Fred was a vice president at Expedia, running product planning for Hotels and Packages for At Expedia, he also participated in various M&A activities, including the acquisitions of, TripAdvisor, and eLong.

Prior to Expedia, Fred co-founded Newtrade Technologies, which grew to become one of the world’s largest connectivity platforms for the hospitality market after it was acquired by Expedia in 2002.



Your email address will not be published. Required fields are marked *

  1. A Semantic Personality | THEMATIX

    […] our data be enriched with personality; i.e., can we craft a romance feed from a big data knowledge […]

  2. Harald Lux

    It’s nice to talk about big data and the post API-era on conferences but for myself (operating a small/medium travel business) I would be quite happy to see OPEN APIs become common sense in the travel industry first/finally!

    By the way: Amadeus is still requesting to sign a non-disclosure agreement to even look at their API specifications.

  3. Tudor Cobalas

    Nursing big data and knowing the work around on how to get the relevant data for businesses, this is the future. The abundance of information and data coming from social networks, media, publishers, bloggers etc will push new models of business, of big data filters that can serve the relevant info.

  4. David Thomson

    I was lucky to be there in person to watch the exchange. For all those that were not in attendance, imagine a room full of API providers, a team from Amadeus, a rep from Travelport, the CEO and a Chair from OpenTravel Alliance, and some developers. It was quite a scene and Frederic is a brave man.

    After writing a meta search for two years, my take on this is that the API providers haven’t adapted to the consumer search market and are still just serving travel agencies. This makes the APIs generally suck at providing us the data points that we actually want. The side affect of this is massive xml bloat and lack of speed associated with the SOAP responses which are loaded with a bunch of irrelevant data. This is the stuff that is needed to book a trip and isn’t affecting the job of the travel agent on the other end with the extra 1-2 seconds of response time. Not so for Joe WebSurfer.

    This leads me to believe that the providers should be creating Lite or new versions of their APIs to adapt to this market. And they should make them easier to consume.

    For instance give me the ability to search for fares for specific dates but also give me the option to check for just the lowest fare for two days before and after the dates I choose. Right now that takes us 5 very slow SOAP requests. Give me 1 slimmed down and fast response for 2/5ths the cost of the 5 responses. I’d bet that we’d all happily pay for that.

    There have to be hundreds of exotic uses cases and data points like this that would help make consumer travel searching better. This is the type of magic we’ve all be waiting for and Its about time for the big data providers to figure them out and provide them as a set of APIs.

    On a side note, I’d like to commend what I saw of Amadeus Labs via Roman from Deal Angel’s demo. Sometimes providing an API that is “blazingly fast at returning next to nothing” is exactly what we need. Perhaps in the next version they’ll figure out the other data point he wanted which was median price instead of the 180 calls to the API.

  5. Bruce Rosard

    Fred – this is a great post that is, in many ways, one of the most interesting perspectives of Google Flights I’ve seen. Will you be at PC11? I can’t believe we haven’t met yet, and look forward to the opportunity.

  6. Timothy O'Neil-Dunne

    Finally we see a dawning of the determinants of change. Travel has big data. Travel abuses big data.

    However this is not really a technology problem. We have good quality technology – as Fred points out a lot of it in the Open Source arena – that handles big data. So why dont we have more applications and services using big data tools?

    I wish someone from one of the legacy GDSs would actually publish a picture of the cabinets in which the “mainframe” processors sit. Its scary for some of us who grew up with Big Iron.

    What we have are a number of legacy processes that constrain the data. Rethinking the processes is in my view more important that bringing the technology in. Up till now there has been very little impetus to do that. Now I believe that the demand pull from the user community (particularly for apps) will force the change.

    Resistance is futile.


  7. glenn mcdonald

    A brief but intriguing panel discussion! My rephrasing of your “APIs are evil” comment is “Peephole APIs are short-sighted”. The problem with saying just “APIs are evil”, out of context, is that it’s too easy to assume you mean that there should be NO programmatic data interfaces, when what you’re really saying is that the classic kinds of programmatic data interfaces are too narrow and limited. And, moreover, that this limiting is ultimately not in the interest of the provider OR the consumer, as it only serves to make it hard for interesting things to happen.

    Old peephole way: “You can have any single item you’d like! What’s its unique internal ID?”
    New Big Data way: “Would you like the whole dataset or some subset? In JSON or CSV?”


Newsletter Subscription

Please subscribe now to Tnooz’s FREE daily newsletter.

This lively package of news and information from Tnooz’s web site provides a convenient digest of what’s happening in technology that drives the global travel, tourism and hospitality market.

  • Cancel