Developers grapple with data all the time, so the idea that someone might characterise data as “broken” is probably not going to take many aback.
But, participating in a panel discussion during the THack event at ITA Software offices in Cambridge, Massachusetts, Hopper CEO Fred Lalonde certainly had some of the programmers talking even hours later with his assertions that APIs are “evil” and that “XML is dead, we just don’t know it yet”.
The backdrop for Lalonde’s views is what he believes is a period of tech innovation in the works which may one day be compared to the invention of the wheel and the sextant.
Lalonde believes strides in storage technologies, including cloud computing, and open systems will enable travel — and non-travel — companies to process massive amounts of data, and the end product will be tremendously innovative solutions.
Different parties will be able to merge and interpret this data in unforeseen ways, Lalonde maintains.
Referring to Facebook’s “firehose” of a data feed, Lalonde says companies should seek access to the most amount of data possible, although he concedes the hope is that your computers won’t “blow up” in the process.
APIs are evil, Lalonde says, because they generally provide access to limited amounts of data for individual functions. And, this is a burden to the distribution process, slowing innovation.
In that regard, there is a reason Google’s new Flight Search product doesn’t rely on an API, Lalonde says.
XML’s days are numbered, Lalonde argues, because of its layers of complexity, whereas messaging in this new era of data processing needs to be more “lightweight”.
Of course, there were plenty of developers at the THack who would counter that all technologies die eventually, but XML does have more than a few good years ahead of it.
Other techies on the open-ended panel retold some of their war stories regarding their battles against uncooperative or elusive data.
Todd Williamson, an ITA Software engineer who works on the QPX airfare shopping and pricing system, says years ago ATPCO didn’t have documentation about its fares’ data, but it has “gotten better” and ATPCO today documents about 80% of its airline data.
“Liberating the data from that perspective was very different than it is now,” Williamson says.
These ITA engineers seem to be preoccupied with data liberation.
Glenn McDonald, who works on ITA’s Needlebase data aggregation tool, grapples with the real-time availability of data and explains that the initial step is to access all the data “and liberate it from what it is behind.”
As the THackers in the audience set out to choose which APIs to tap into and to begin work on their 24-hour hacks, the alleged evils, transgressions and liberation of data were definitely on their minds.
NB: Disclosure – Frederic Lalonde is the chairman of Tnooz.
Related posts:
Calling APIs evil because some choose to not make certain data sets accessible is like calling cars evil because people operate them improperly. It’s completely nonsensical to critique the car when the driver is the real problem.
While semantic technologies are likely the future, the “glue” layer of interoperability between systems isn’t going away anytime soon, especially in travel. We can only hope for more APIs and less EDI.
Absolutely agree…
EDI(FACT) is merely a syntax that is structured differently from XML. I’m tired of people slamming “EDI” in favor of XML as if the use of XML also defines the business transaction sets exposed and the data transport and routing requirements. XML excels at making it easier for “developers” to code to, and for allowing “non-experts” be able to read the data. But that comes at the cost of larger messages needing more time on the wire and more work to transform.
The real issue I have with the “Data vs. API” argument I have is that having access to all the data is useless unless you posess the knowledge and power to process it. Airline pricing and inventory data sets are extremely complex and so merely subscribing to the data is not enough. Unless inventory and pricing models are greatly simplified, providing API access to a service that correctly interprets the data is the often better answer, especially for any company that isn’t out to make pricing itself their core business solution, but wants to participate elsewhere in the travel distribution process.
Ironically, like most business standards, those that govern their evolution are those very people who’s jobs would be over if they radically simplified the process…so they never tend to vote for that option..
You have to start somewhere. I understand Fred’s futurist approach, but in certain sectors, all that data is stored on sheets of paper! There is so little data that is actually in state that can be queried, shared, or transferred electronically. I prefer to think of XML as an evolutionary step. XML, mapreduce, protobuf, json, are all means to an end. That end has yet to be written.
Stephen, I agree. However there is a real problem with futuristic thinking – if you look too far ahead – you end up devising something that just won’t get CURRENT market acceptance. Most big changes in the travel industry have come around evolution not revolution. Don’t see that changing any time soon.
I’d like to know what Fred proposes replaces API’s and XML. Facebooks firehose is to all intents and purposes an API isn’t it? Yes many API’s are rate and request limited, but that’s not because they are API’s rather it’s down to the owner of said API.
Advances in computing power will certainly help API’s become more realtime and firehose like but they will likely use similar technologies to the ones used now for data transport and data structure for many years to come.
Having managed many projects to integrate many different data sources into travel sites over the years (from oldschool EDI, to flat files, to XML to Rest etc) and knowing the variation in standards and capabilities of the companies providing them, I think this subject deserves more than just an ‘xml is dead so are api’s’ statement. Would appreciate some ideas… The sector certainly needs some.
Some of my clients are just adopting XML. Others are years away from being able to do so… The struggle to integrate and make sense of many types of data sources in online travel isn’t going away any time soon I fear.
valid points, steve
a very valid point indeed
I think Mr. Lalonde either doesn’t know what an API is (unlikely) or is using API to describe a currently fragmentted and non-standard ways of accessing data. It sounds on the surface as if Hopper is using data feeds/APIs to populate their travel database into a concentrated source for customers. This would allow for faster querying and quick cross-correlation vs multiple external searches done through APIs. So perhaps his poorly worded or poorly quoted statement is that the current implementaiton and use of APIs is evil. Perhaps he doesn’t like XML due to the overhead?
maybe he is referring to developments with semantic markup – but for sure he’ll be chiming in to the debate
I hope Mr. Lalonde never experiences true evil directly. He should probably read more.
Sigh, XML is a solution. Lets stop mixing up business requirements with solutions.
Where Frederic is right
* Lightweight data transfer – I have been pushing for lightweight data transfer standards for years. Must dig out some blog posts from my old blog on this matter. Sadly Open Travel only interested in server to server communication standards which are rarely useful for web based systems.
* Most APIs (whatever they are presented in) only support the functions the designers intended. i.e. most travel APIs are provided on the assumption that the remuneration will be based on transactions.
Where there is some room for discussion
* Often the data presented to a “consumer” (meaning someone taking the data, not a traveller) is related to who the consumer is. Authentication is required. Prices and sometimes availability will be different depending on who is accessing the data. You can’t just publish a public feed….. its more involved than that.
Where Frederic is wrong
* We provide a firehose of date, price, availability data for 120 + tour operators. We use XML. We also have an out of the box JSON and JSONP mechanism that can be used……. the data format is 100% irrelevant.
Good debate though (boss)
I am very sure that there will be a Google Flights API and there will be so many excellent mash ups based on it that Hopper’s “sub-one-second” target will just be a standard that many sites will have access to in sub-10-ms via Jsonp/Json without $8 mio venture capital and 600+ servers.
What is the status of Hopper’s “The company views itself as a B2B alternative to Google-ITA Software” vision (http://www.tnooz.com/2011/03/14/news/hopper-ready-to-unleash-full-text-travel-search-system/)? Is this still valid or dead like XML?
Btw, the only thing that is evil and close to dead in the web era is a crazy ego name dropping ratio of 8 times within 8 sentences.
Could have been a great idea to begin by giving the definition of an API ….
In the travel industry we had to deal with several revolutions, all of them linked with technology standards and solutions. For me, and I agree with Frederic, API is more or less linked to the two main first evolutions of the travel industry:
1) 1st generation – GDS era: verb based API, sending commands to remote systems. Used extensively by GDS to open up their solution. Verbs were basic at the beginning and then were extended to offer more “intelligent” requests (so less application logic to be coded on user side). XML was not mandatory … The “exchange dialects” was more or less based on the remote systems data … or the GDS business philosophy and CRSs level of integration.
2) 2nd Generation -OTA and Internet era … It was the time of REST vs. WS, SOA, XML dialects and the long tail. These API were built to ease client side development, to facilitate crawling by search and meta-search engine and reduce cost of integration. The web standards were used for point to point communication (HTTP, HTTPS, SSL, etc.) but also for easy tracking, invoicing or booking referral … Data model were built to be traded and generally not shared for free … The XML dialect used was a mirror of the commercial vision and the data quality and completeness. That’s why, for me, OTA was a failure (I know that 5 people in the world are using it and will claim that it is a success).
3) 3rd generation – Travel 2.0: data deluge (social, open data), direct connect, data aggregation at low cost and cloud computing. The dream now is to be able to gather and use non structured data with structured data to improve trip planning, booking, in REAL TIME (push/pull/notification/event based). In this world, the notion of API is a nonsense. I see it more like interacting with an expert system.
First of all, thank you all for great commentary. Seldom seen these days. Then I would urge you techno people to perhaps suggest and submit standards to http://schema.org/ – I find it to be lacking in the specific needs for the travel industry. If a common standard can be set for Structured Markup Language (there’s 5 other names for semantic markup floating around) for the travel industry I think we could be well on the way.I might be wrong but it seems they are working with W3C on this.
No standard will ever be created in this industry … The OTA XML schemas already exist, but nobody really cares. There are giants travel data distributor and they dictate the “technology”. Now every travel suppliers (air, and soon hotel) will push its own standard.
_XML is Dead_?
No more than FORTRAN or COBOL are — they’re still being used, look it up — but our experience is that (with the notable exception of bureaucrat-heavy organizations) most data transfer has moved to JSON.
The movement from XML to JSON is being driven by one simple thing: JSON is much easier to use, deploy, understand and consume. Yes, there is a lot of power to express information in XML but in fact that has a direct correspondence to “programming complexity” and we’re all under a lot of pressure to ship _now_.
_APIs are Evil_?
If I was planning to write the most powerful program in some space, yes, it would involve a program talking to a database in the same server stack. How does that program talk to that database? Whoops, an API. Perhaps that doesn’t count. Then I controlled everything, I’d do a kick ass front-end using HTML5, maybe something like http://www.google.com/flights/. How would that get its data? Ah … an API. If you’re using Firefox, turn of Firebug and watch all the XHR messages fly back and forth (using JSON!) making this all work.
What then is evil with APIs? They are the glue that holds everything together in any software stack, and conceptually they’re the best way to manage complexity. Every large system has to use them otherwise things rapidly become unmanageable.
There is no programming “silver bullet” — what the best API is going to be is defined by use. If you know the problem exactly then you’ve probably figured out the best API for that problem. Knowing the problem exactly is usually isomorphic to “shipped”. Before that, it is all guesswork and fine tuning.
But hey, for 95% of applications, what the Twitter, or the Facebook, or the Whatever API does almost exactly the trick you’re looking for.
I was actually there when Fred said what he said. The discussion was about ‘Big Data’, No SQL databases, pentabytes of data, etc. And in the given context XML APIs do appear a little limited; I fully support Fred on this one. Particularly the XML approach has been criticized, not APIs as such. For example, probuf has been suggested as a healthier alternative for building high volume data interfaces. And come on, it was a rather ‘I Have a Dream’ exchange of ideas which didn’t really call for API genocide!
At DealAngel we process millions of data points a day to calculate fair market values of offered hotel deals and I can vouch for APIs being the narrowest pipes in our technology. They are meant to provide a certain volume of information and are limited to their ‘intended use’ only. The way I see it, they limit innovative uses of information. We often have to request/download loads of unnecessary data just to extract a tiny bit of information from a cross-query on this load of data.
Now, speaking of futuristic stuff, I can imagine that at some point APIs will be more intelligent and allow us to run custom queries on the API provider side, including statistical and grouping functions. For example in my THack project I needed median hotel rates for each day in a particular destination for the next 180 days. In order to do this I needed to hit the Amadeus API 180 times, get all hotels rates then calculate the medians. I repeat: the API had to be abused and hundreds times more data has been requested and downloaded than needed. To my great pleasure Amadeus rocked – everything worked blazing fast. But wouldn’t that be nice to request the medians to be calculated on the API provider side instead?
Again, for many simple tasks such as syncing itineraries or submitting a booking, APIs the way they are now are perfect. But for data mining and semantic applications we are going to need something different. Something else. Something much more intelligent. What this will be is subject to discussions and innovations which I am hoping to be a witness to and a part of.
There are only three possibilities:
(1) you write something that’s effectively programming on the server side
(2) you replicate the data using a firehose
(3) you have APIs that do exactly (or pretty close) to what you need.
If your problem space was “Twitter”, in which case:
(1) you’d have had your app broken (say) 10 times and rewritten 10 times as Twitter changed their data model and infrastructure to cope with the increasing complexity of the problem space
(2) you’d have to be close to as clever as Twitter in deal with the volume of data coming in, plus you’d probably using scads of bandwidth for information that you will never use
(3) depends on the provider having the exact issues in the problem space as you do.
It gets easier when the problem stabilizes, because there’s better a better conceptual model of what the issues are. However, most of the interesting cases are where we don’t even know what the issues are going tobe in two years time.
Nice discussion here , I think it is just to scare competitors, Eating up the small fishes. unless the quantum transportation becomes a reality