Long Live the Web

[Pangloss]

Interesting opinion piece by Web creator Tim Berners-Lee in the new Scientific American.

http://www.scientificamerican.com/article.cfm?id=long-live-the-web

 

He goes on at great length about the importance of Net Neutrality and the dangers of commercial Web sites like Facebook that don't use open standards.

 

Facebook, LinkedIn, Friendster and others typically provide value by capturing information as you enter it: your birthday, your e-mail address, your likes, and links indicating who is friends with whom and who is in which photograph. The sites assemble these bits of data into brilliant databases and reuse the information to provide value-added service—but only within their sites. Once you enter your data into one of these services, you cannot easily use them on another site. Each site is a silo, walled off from the others. Yes, your site’s pages are on the Web, but your data are not. You can access a Web page about a list of people you have created in one site, but you cannot send that list, or items from it, to another site.

 

The isolation occurs because each piece of information does not have a URI. Connections among data exist only within a site. So the more you enter, the more you become locked in. Your social-networking site becomes a central platform—a closed silo of content, and one that does not give you full control over your information in it. The more this kind of architecture gains widespread use, the more the Web becomes fragmented, and the less we enjoy a single, universal information space.

 

He makes a number of excellent points, but I think he ignores one that I thought might be interesting to discuss: Is it possible that the existence of BOTH open standards AND proprietary standards enables more ingenuity and development than would be possible with either of those ideological methods alone?

 

I am all for most of what he's saying -- Net Neutrality, and the concept of regulation in general, are perfectly fine subjects for consideration and, where appropriate and after fair and open debate, implementation. What I'm wondering about is this underlying argument that "open is better". In a sense it's become a kind of second-tier competition between open and proprietary, with each having its own motivations and rationales.

 

Maybe that's what we want. We'll take the benefits of both, and we shouldn't be afraid to step in when necessary and take some of the proprietary stuff away when it's best. By the time the people step in the entrepreneur has already made a fortune anyway, so you aren't killing that motivation. And in some cases we might even shy away from promoting an open standard because the need isn't quite there yet, allowing the proprietary types to create the need.

 

What do you all think?

[Dak]

Is it possible that the existence of BOTH open standards AND proprietary standards enables more ingenuity and development than would be possible with either of those ideological methods alone?

 

Facebook doesn't use any standards. to my knowledge, there's no 'export data' function which could aid migration to another Social Network, nor an 'add a friend on myspace' feature that results in facebook and myspace communicating with each other in order to defragment the SNs, nor a 'sync with another SN' feature, all of which would be enabled by a universal [open|proprietary] standard.

 

as for open vs. proprietary standards, what's that actually mean? it can't be open as opposed to closed, as that'd imply that proprietary standards == closed standards which makes no sense at all (secret standards just wouldn't work)... whilst open as in 'anyone can change them' also somewhat conflicts with the concept of a standard... unless some group has the power of Having The Last Word, in which case i suppose the difference between open and proprietary standards would actually be quite slim...

[Cap'n Refsmmat]

What exactly do we mean by a "proprietary standard"? Most things we call "proprietary" are closed systems; that is, there's no standard for anyone to attempt to adopt, because the company keeps such information secret. Systems like Windows Live Messenger (aka MSN Messenger) are proprietary standards because Microsoft has not released the details of the communications protocol.

 

Is that what you mean by "proprietary standard"? Or do you mean a standard that is released but subject to license terms? I mean, it's not entirely true to refer to a specification that is never released or specified as a "standard"...

[Pangloss]

Well okay "proprietary formats", then. It's not really standards I'm trying to get at here, so much as economic ideologies.

 

The thing I felt that Berners-Lee might be overlooking is the motivational nature of private enterprise. He's not arguing against it, of course, he's pointing out the benefits of open standards, which is all fine and good. I just wondered if it's really the combination of the two that makes the Web so powerful.

 

His perspective is understandable because it was the openness of the HTTP standard that made the Web grow so rapidly. But a lot of what's great about the Web today is commercial in nature. It's often paid for with advertising instead of subscription or usage fees, but the money motivation is still there. If memory serves, Google makes over $16 billion per year on advertisements on its search results pages. Facebook founder Mark Zuckerberg is worth almost $7 billion.

 

But "free" innovation isn't dead either. Many of my favorite places on the Web are non-profit forums like this one. The Wikipedia wasn't launched until 2001, and took a few years to really take off and catch people's attention; now we can't imagine a world without it. And nary a profit motive in sight -- a Wikipedia with pop-up ads is inconceivable.

 

I guess what I'm wondering is if the Web is kind of a microcosm of hybrid economic ideologies -- socialist and capitalist. It wouldn't work without both of them present and contributing.

 

Kinda like Western civilization as a whole.

[JohnB]

I don't see a problem, the web is working like business always has.

 

WRT open/proprietry it's similar to cars. Cars are built to a given set of standards in each nation so those standards ore "open". Those standards don't require General Motors to disclose to Ford the codes of their engine management systems or their client list of who bought a GM vehicle last year, so that part is "closed".

 

Nor is there any reason to expect to be able to "import" a Ford part onto a GM vehicle. Why should the net be different?

 

(Net neutrality is a totally different question.)

[Pangloss]

Makes sense to me.

 

BTW, I thought engine management codes were open under the OBD-II standard? I actually bought one of those hand scanners recently after my engine started misfiring, wondering if it might tell me if it was just bad gasoline or an engine malfunction. Turned out to be worthless -- the code just translated to "your engine is misfiring". (rofl) Kind of a fun gadget to play with, though, and it's cool that you can plug it into any car.

[D H]

This very forum depends on proprietary formats. That doesn't stop web crawlers from crawling in. At this very moment, this site is infested with spiders from Google, Yahoo, Ask Jeeves , ...

[Dak]

This very forum depends on proprietary formats. That doesn't stop web crawlers from crawling in. At this very moment, this site is infested with spiders from Google, Yahoo, Ask Jeeves , ...

 

Plus, it used to be vBulletin, another prorietary system: yet we could still migrate to IPB, and can still migrate to, idunno, phpBB or something. can even copy/paste from one forum to another with relatively few problems, and can certainally hyperlink from a vB forum to an IPB forum.

 

That's not really analogous to facebook et al, and I think that's Berners' point...

 

So:

Well okay "proprietary formats", then. It's not really standards I'm trying to get at here, so much as economic ideologies.

 

tbh, I don't think you can seperate this from standards. without considering standards, I think OS and proprietary go well hand-in-hand. I mean, i can code just about well enough that'd i've modified a few programs before so i suppose technically I'd prefer it to be OS in case i need to fix a simple bug, but tbh as long as it's free and cool i don't really care what economic forces drove it's development.

 

Problem with proprietary is when it'd be better for US if the companies shared and inter-operated, but more profitable for THEM if they didn't, so they don't; which not only sucks but also removes the burden of innovation in order to profit and causes vendor lockin.

 

see, e.g., openoffice vs. ms word for example. Both have innovated in the field of word processorage, and it'd be perfectly OK that they both exist: except that, due to the fact that MSW is proprietary and how MS have (ab)used this fact it's genuinally hard to chose OO over MSW.

[Pangloss]

see, e.g., openoffice vs. ms word for example. Both have innovated in the field of word processorage, and it'd be perfectly OK that they both exist: except that, due to the fact that MSW is proprietary and how MS have (ab)used this fact it's genuinally hard to chose OO over MSW.

That's interesting. How has Microsoft abused the proprietary format of Microsoft Word to create that situation?

[Cap'n Refsmmat]

That's interesting. How has Microsoft abused the proprietary format of Microsoft Word to create that situation?

Migrating to any other system is nearly impossible, because converting documents to any other format is exceedingly difficult.

 

Office Open XML changes some of that, but it's a gigantic standard and Microsoft products do not fully comply. They have quirks that cannot easily be converted.

[Pangloss]

Sure, I agree with that, but I don't think that's really where the "abuse" comes into it -- Microsoft has a far shadier and more malignant history than that. The retail shelf-space wars of the early 1990s may have had more to do with Word and Excel's market dominance than proprietary file formats, and at that time all the office-app programs had proprietary formats -- there were no open standards. Once they won that head game with CEOs and CTOs, Lotus and Word Perfect were done. Then XML comes along.

 

Also it's unfair to call proprietary technology "abuse" when the approach creates a market that didn't even exist. Apple is going through the same thing now with the AppStore. People now complain about it being closed and level words like "abuse" at it, but three years ago nobody was buying any apps for smart devices at all. I do think it's appropriate for society to decide after the fact that a system may need to be forced open, but we can at least recognize achievements and not demonize them. I think it's only when a company actually breaks the rules (laws) that words like that should be leveled.

 

That having been said, just because a government requires an open file format (for example) doesn't mean that a company can't innovate. We can require future development to use that approach and companies can still make money. But that's just file formats and other familiar technologies. New innovations like the AppStore may be proprietary in a different way. We shouldn't just blanketly say "all companies must use open standards all the time in every possible way", lest we stifle the profit-driven side of innovation. Better to let companies do their thing, make a little scratch, and THEN come in and say "okay, that's going to be an open standard now".

 

In other words, the word processor history is working out about right. It makes sense now for governments to step in and require open file format standards. At some point it may make sense for governments to step in and mandate open "app stores" in mobile smart devices. We take these things as they come, and keep the system flexible and supportive of innovation.

[padren]

When talking formats and engines, SQL comes to mind as a pretty big mix of holy-grail level achievement of standards and at the same time painful idiosyncratic derivations.

 

Just the fact that you can hack together a database in MS Access and actually run it on a production website (NOT that such an act would be encouraged) and connect through any number of proprietary or open source application layers is amazing, when you look at everything from MySQL to Oracle in terms of engines you have just about everything that can happen, happening all over the place at once.

 

When you take into account the complexity of atomic transactions, indexing, data integrity and all the requirements for production level databases (plus the need to constantly innovate and one-up the competitors) the cross-compatible open standards make anything in the consumer market (browsers, office doc formats) look pretty flimsy by comparison.

 

 

 

Btw, good points on the app-store Pangloss. When a new proprietary technology creates a market, it's hard to fault the company for not opening the doors wide open. It was a non-issue when Atari game cartridges came out and no one accused them of being closed off because it was so hard to access just because of the technology limitations of the time. Since we've moved to a point of having almost virtual hardware (touchscreen keyboards, UIs) all it takes to extend something is data, so people are painfully aware of how much better something could be if they could get under the hood.

 

That said, there is something about blocking what you can and cannot do with hardware you own (warranty aside) that bothers me about crippleware. If you own the equipment, you should legally be able to do anything from nothing at all to smashing it with a hammer.

[Pangloss]

That's a great example with SQL -- an open standard with lots of room for profitable innovation.

[swansont]

Btw, good points on the app-store Pangloss. When a new proprietary technology creates a market, it's hard to fault the company for not opening the doors wide open. It was a non-issue when Atari game cartridges came out and no one accused them of being closed off because it was so hard to access just because of the technology limitations of the time. Since we've moved to a point of having almost virtual hardware (touchscreen keyboards, UIs) all it takes to extend something is data, so people are painfully aware of how much better something could be if they could get under the hood.

 

That said, there is something about blocking what you can and cannot do with hardware you own (warranty aside) that bothers me about crippleware. If you own the equipment, you should legally be able to do anything from nothing at all to smashing it with a hammer.

 

Atari was more closed than the current situation; you can write iphone/ipad apps if you can program. Was that true about Atari game cartridges?

 

As far as open/closed systems, every commercial company is closed in the area(s) where it makes its money. Besides, I don't necessarily want information I give to facebook to be available to everyone on the web, and I don't want someone else deciding that it should be.

[padren]

Atari was more closed than the current situation; you can write iphone/ipad apps if you can program. Was that true about Atari game cartridges?

The technology was difficult enough to utilize that open standards would have been moot - it took a lot more than writing data to build a game for the Atari console, even if you knew the standards. Technology wouldn't even advance enough for that question to be raised for some time.

As far as open/closed systems, every commercial company is closed in the area(s) where it makes its money. Besides, I don't necessarily want information I give to facebook to be available to everyone on the web, and I don't want someone else deciding that it should be.

Ogre3D is an A list quality 3D rendering engine written in C++, fully open source and has been used in delivered, A list games. The engine was originally started by Steve Streeting, and while much of his contract work has been closed due to non-disclosure contracts, it's the fact that his product is open source that allowed him to do a lot of contract work as an expert with that engine.

 

There are many other examples of this sort of model, but it's worth noting that his revenue is based on an entirely free-to-use and free-to-get/change open source product.

[swansont]

The technology was difficult enough to utilize that open standards would have been moot - it took a lot more than writing data to build a game for the Atari console, even if you knew the standards. Technology wouldn't even advance enough for that question to be raised for some time.

 

But that just means the barrier was different. It was still effectively a closed system, with only a handful of developers able to produce games.

 

Ogre3D is an A list quality 3D rendering engine written in C++, fully open source and has been used in delivered, A list games. The engine was originally started by Steve Streeting, and while much of his contract work has been closed due to non-disclosure contracts, it's the fact that his product is open source that allowed him to do a lot of contract work as an expert with that engine.

 

There are many other examples of this sort of model, but it's worth noting that his revenue is based on an entirely free-to-use and free-to-get/change open source product.

 

If it's free-to-use and free-to-get/change, where does the revenue come from?

[padren]

But that just means the barrier was different. It was still effectively a closed system, with only a handful of developers able to produce games.

It was effectively a closed system, but the fact it existed before an open system could even be realistically imagined changes the nature of the beast a bit, since there was no alternative to the closed system.

 

Before certain technologies existed, it was impossible to record audio. That doesn't mean all creators of audio performances adhered sternly to a "no recordings" policy - the question wasn't asked because the very thing the question addressed did not exist yet. The net effect is about the same, but the distinction is worth noting.

 

If it's free-to-use and free-to-get/change, where does the revenue come from?

If you want to use the engine in a commercial project, you can for free - but even though it's very well documented, the expense of training programmers to learn it inside out to the point of creating niche proprietary project-specific extensions is much higher than paying the creator or a veteran contributor to head the development.

 

The revenue comes from the value of the expertise.

Edited November 25, 2010 by padren

[swansont]

If you want to use the engine in a commercial project, you can for free - but even though it's very well documented, the expense of training programmers to learn it inside out to the point of creating niche proprietary project-specific extensions is much higher than paying the creator or a veteran contributor to head the development.

 

The revenue comes from the value of the expertise.

 

So it's basically advertising. Which is my point — parts of the system that do not matter are open, and the part that makes the money is not.

[Dak]

Sure, I agree with that, but I don't think that's really where the "abuse" comes into it

 

I was under the impression that they keep changing their format for no good reason, thus forcing the (probably incapable of keeping up) competitors to redo all their compatability work.

 

Also, there's the whole OOXML vs. MS-OXML thingy...

[Cap'n Refsmmat]

I was under the impression that they keep changing their format for no good reason, thus forcing the (probably incapable of keeping up) competitors to redo all their compatability work.

 

Also, there's the whole OOXML vs. MS-OXML thingy...

Well, since the format was closed for many years, they could change the format for any reason without explaining it to anyone else. It's impossible to know what the true motives were.

 

And yes, there were allegations that Microsoft representatives bought votes on the ISO standardization committee so Office Open XML could become a standard.

[Pangloss]

I'm not sure exactly why these standards arguments matter -- I don't really understand the details of it. What I think it boils down to is that if Microsoft is saying that they want to store data that a user has created inside a file, and the format doesn't allow that kind of data to be saved, then I would think that they have a legitimate complaint about the standard. But if the format does allow it, and they're lying in order to stifle competition, then that's another matter entirely.

[Cap'n Refsmmat]

I'm not sure exactly why these standards arguments matter -- I don't really understand the details of it. What I think it boils down to is that if Microsoft is saying that they want to store data that a user has created inside a file, and the format doesn't allow that kind of data to be saved, then I would think that they have a legitimate complaint about the standard. But if the format does allow it, and they're lying in order to stifle competition, then that's another matter entirely.

That's not what the argument is. For years, until Office 2007 came out, Microsoft Office used a proprietary binary file format to store its documents. There were no published standards for how to read the formats, they changed frequently between Office versions, and competitors wishing to read Office documents had to reverse-engineer the files to determine how to read them. This is why OpenOffice and others have a hard time preserving formatting when opening old Office files. Microsoft's market dominance meant that companies could not easily move away from Office, since no other software could open Office files.

 

Microsoft then put together Office Open XML, their new "standard" XML-based format for Office documents. The standard is several thousand pages long, and Microsoft's own products produce documents that violate the standard. Writing software to read the format is difficult due to the standard's length and vagueness in places; I believe parts of the standard had settings for, say, "emulate Word 97 page break behavior" without defining what Word 97's behavior was, as one example.

 

Office Open XML was then accepted as a standard by ISO, although there were allegations that Microsoft paid off standards committee members to get their format standardized. OpenDocument, a competing standard developed by OASIS, had already gotten ISO standardization.

 

The issue is less important now, since Office 2010 more strictly follows the ISO standards, and other software makers are adding Office Open XML support to their products. There is significant pressure on Microsoft to follow open standards now that many governments have made a commitment to open standards -- else they may find they can't read their own important documents in ten or twenty years.

 

This very forum depends on proprietary formats. That doesn't stop web crawlers from crawling in. At this very moment, this site is infested with spiders from Google, Yahoo, Ask Jeeves , ...

 

I think the important part to Berners-Lee is that nearly all of the data from this forum can be extracted easily. Each post has its own URI, and since we serve (nearly) standard XHTML, any interested party can scrape all of our posts, discussions, profiles, and so on. The software also provides features to let you download your personal conversations, for example, and it wouldn't be hard to write a robot to save all of your posts.

 

The same can't be said of Facebook, where you can't easily scrape data. Not that you'd want to, anyway, considering how inane it all is...

 

This forum isn't particularly proprietary, either; you could say that the database schema is proprietary, since it's commercial software, but the database uses MySQL, which is open-source, and I can inspect the database schema with any free utility. In fact, I often debug issues by looking through the raw data. We serve (nearly) valid XHTML and CSS, specifically optimized so that search engines can scrape the text easily. There's not really a proprietary file format behind any of SFN.

[Pangloss]

I understand all that, the part I don't get is the practical side. Why does this matter to users, or, say, third-party developers?

[Cap'n Refsmmat]

If you're a user, you can't use anything else than Microsoft Office. You can't switch to another competitor or use third-party software to manage your Office files, no matter how good it might be. If you're a third-party developer, you're extremely limited in what you can do with Office files.

 

As an example: My mother is a professional translator. There is a wealth of software to aid translators; most systems store phrases and their translations, and suggest an old translation when a similar phrase is found. (This is exceedingly useful in repetitive texts, like computerized phone menu systems or pharmaceutical brochures.) Such software has to interface with Microsoft Word so the translator can use it while editing a Word document. However, in our experience the integration can be terrible -- the translation software has to insert formatting to keep track of its progress, but it interacts badly with Word's unpredictable proprietary formatting and often results in a mess. If the Word format were standardized, computer aided translation could be easier.

[Pangloss]

Thanks.

 

I agree that was historically the case, but is that entirely true today? I thought you could open and edit Word and Excel does in Google Docs (I think they just added editing recently) and in that standalone Office alternative, which I forget the name of.

 

The translation example sounds like it's not really helped by this, of course. That sounds like an interesting opportunity, actually.