Beating the algorithm, for now

Screen Shot 2014-07-03 at 11.32.33 AMThe New York Times has a fun interactive that lets you compete against an algorithm designed to predict which tweets will get the most retweets. The Times also has a story about the algorithm and its implications. Its takeaway:

That an algorithm can make these kinds of predictions shows the power of “big data.” It also illustrates a fundamental limitation of big data: Specifically, guessing which tweet gets retweeted is significantly easier than creating one that gets retweeted.

Sure. But predicting which of two phrasings will get the most retweets is still plenty difficult, and as part of my job is social media, it’s totally a “skill” I supposedly have.

So I was pleased to beat the algorithm 18 to 13. But even in victory, it’s clear that this is a losing gambit for me, over time. First off, there’s a chance I just got lucky, especially since the story says the algorithm usually guesses right 67% of the time. (I’m not sure how the interactive works, and whether they picked a single set of harder examples, if mine were different from others, or what.)

But any case, even if I wasn’t lucky, there’s a single question that to me is the bottom line in terms of humans’ race with machines: which of us do you think will get better faster?

If you do this same kind of test in two years, how much better will I be? Sure, maybe I’ll have improved a bit (although maybe not). But with more data at its disposal, faster processing power, new statistical techniques, etc. the room for an algorithm to improve is far greater.

The same holds for many other kinds of forecasting. That humans still beat algorithms at predicting, say, geopolitical events (I’m making this up) is interesting. But even as we get better, our progress is incremental, linear. It’s the algorithm that’s poised to win most improved.

Our patent problems go way beyond trolls

UPDATE: More recent data documents the serious uptick in patent troll litigation. Likely still true that the patent problem goes way beyond trolls, but they are a problem nonetheless. Recent research is here.

I did a Google Hangout with two intellectual property experts this week, and wrote an article to go along with it. The jumping off point was Tesla’s patent sharing announcement, but really it ended up being broader than that, covering the problems with our patent system and the possibility for reform.

One thing it was not really about was patent trolls, and it occurs to me based on some of the reaction to the article that I should have made this more explicit.

Here’s the chart from the post showing the explosion of patent litigation in the U.S.:



(If you’re curious about that spike at the end, read the update at the bottom of my post.)

The consequence of this dramatic increase is that patents have the effect of making innovation less profitable, rather than more so, in all industries except pharma and chemicals. In other words, when you count up the benefits to innovators from excluding others from their invention, and then subtract the cost of litigation, you get a negative number.

There are many reasons for this, among them the fact that in industries like software the “boundaries” around patents aren’t clear. So you have a patent and I have a patent and neither of us are quite sure what either of the patents does and doesn’t cover. That leads to a lot of unnecessary litigation, and beyond that just a lot of uncertainty.

But it’s worth spelling out that while patent trolls are a problem — one that needs to be addressed — they are not the primary driver of this explosion in litigation. Much of my post borrows from James Bessen of BU, one of the experts I interviewed, who has done research on this question. Here’s what he says in his book Patent Failure:

We also considered the role of patent “trolls,” which we define narrowly as individual inventors who do not commercialize or manufacture their inventions. One story claims that the increasing availability of patent litigators willing to work on contingency fees has spurred lawsuits by such trolls, who might otherwise be unable to afford litigation. The share of lawsuits initiated by public firms has not declined, however, nor has the share of lawsuits involving patents awarded to independent inventors increased. This suggests that the increase in litigation cannot be mainly attributed to patent “trolls,” at least through 1999. Of course, if we use a broader definition of “troll” that includes all sorts of patentees who opportunistically take advantage of poor patent notice to assert patents against unsuspecting firms, then troll-like behavior might be a more important explanation. Indeed, if patent notice is poor, then all sorts of patent owners might quite reasonably assert patents more broadly than they deserve. But then it is more appropriate to attribute the surge in litigation to poor patent notice, not to trolls per se.

As indefensible as the business model of companies like Intellectual Ventures is, that pure troll model does not itself explain the rise in patent litigation.

I wish I’d made this point even in passing in my HBR piece this week. It’s easy to blame the trolls, as well we should. But our patent problems go well beyond them.

Be Xerox not Apple

Having just finished the Steve Jobs bio, and so freshly reminded of how Apple borrowed the idea of the graphical user interface from Xerox to great effect (only to later turn around and call Microsoft a thief for doing the same), I enjoyed this back and forth on Xerox. It’s a piece of an extremely long Q&A between Felix Salmon and Jonah Peretti about… everything.

JP: The thing is, if you care about having an impact on the world, the too-early mode is the highest leverage point because you can have an idea, build a mock or a prototype of it, and then have those ideas find themselves in products that other people build that then scale up to massive.

FS: You’d rather be Xerox than Apple?

JP: People always talk about Xerox as a sad story.

FS: Maybe.

JP: I mean some people do. If all you value is money, then it’s a sad story. But if you think that the graphical user interface is a cool thing and you worked on the graphical user interface at Xerox, you can feel like, “This has a big impact on the world.”

FS: It’s a way of looking at the world through a lens of capital rather than labor. I’m sure the people who invented the graphical user interface at Xerox are doing very well for themselves right now.

JP: Right. And some of them have a sense of personal satisfaction that they had a big impact, even if they didn’t profit from it at the scale that they could have.

The history of the tech industry is full of case studies where we ask why some once great company couldn’t capitalize on a breakthrough it had created. It’s a fascinating question, but it’s worth remembering that it’s a question for firms more than it is for people. It’s easy to see why a firm would want to capture some of the financial value of a breakthrough, but for many people that’s not necessarily the goal. Sure, some great technologists truly are motivated by getting their innovations into as many hands as possible. But not all of them.

Sometimes, as Peretti says, the true impact gets made by the innovators who are too early, while the riches go to those who follow later. For lots of people, it’s the impact that matters.

Net neutrality is about more than small vs. big

tuxWith the FCC reportedly considering allowing paid “fast lanes” for internet traffic, the principle of net neutrality looks more at risk than ever. One of the big concerns of net neutrality advocates is that its absence might empower incumbent firms over newer, smaller, more innovative ones. That is a very valid and important concern.

But small firms represent only one sort of innovator, and arguably not the one most at risk from pay-to-play operations.

Here are a couple examples of the protect-the-startups meme in recent coverage. From NYT:

Consumer groups immediately attacked the proposal, saying that not only would costs rise, but also that big, rich companies with the money to pay large fees to Internet service providers would be favored over small start-ups with innovative business models — stifling the birth of the next Facebook or Twitter.

And here’s an editorial from The Financial Times, arguing that net neutrality may no longer be the right goal:

The fine detail of the FCC’s decision will matter. The regulator will have to ensure its reforms do not create barriers to entry for small and innovative companies – the internet giants of the future.

At The New Yorker, net neutrality advocate and media scholar Tim Wu goes a bit broader:

We take it for granted that bloggers, start-ups, or nonprofits on an open Internet reach their audiences roughly the same way as everyone else. Now they won’t.

To the extent that we can protect innovative new firms from being crushed by incumbents before they get off the ground, that’s great. But the next Facebook or Twitter, while at risk, also has the ability to raise capital and spend it on faster content delivery. (The details of the FCC regulations aren’t yet clear but it sounds like there will be a requirement that similar pay-to-play offers be available to all comers.)

An even bigger risk then is for non-professional content producers and for peer-to-peer, commons-based production. The bloggers Wu mentions could fall into this category, though if they’re using a proprietary platform like Tumblr or Medium they might not. A peer-to-peer project like Wikipedia has its nonprofit arm, but little ability to raise the capital necessary to ensure delivery. Less organized peer production efforts would be at even greater risk. A distributed network of independent bloggers might produce great content, but that content will be delivered slower than content produced by professionals, or by amateurs who’ve bought into a commercial platform. Suddenly, peer production is at a huge disadvantage relative to commercial production unless it has the weight of a commercial enterprise behind it.

The promise of large-scale production outside of firms or governments, from open source software to Wikipedia to independent blogging, was once one of the greatest promises of the internet. And it is even more at risk from the legalization of pay-to-play than are startups. Sure, incumbents might lean on startups who can’t afford to pay for faster delivery. But just as worrying is the thought that startups might raise venture capital to pay for faster delivery in order to crowd out commons-based peer production.

The net neutrality debate isn’t just about small vs. big. It’s also about commercial vs. the commons.

The promise of NYT Now

NYT now

There are few if any media outlets that can really go up against the big social networks and have a prayer of stealing away attention. The New York Times might be an exception.

When I first heard about NYT Now I didn’t think twice. It seemed like yet another addition to an already complicated, expensive offering. And its name suggested the reason I didn’t need it: speed is not the primary thing I’m looking for in consuming the Times’ content.

But a piece at Nieman Lab has me rethinking my skepticism:

NYT Now can be seen in part as an Empire Strikes Back play: It aims to take readership back from Twitter and Facebook.

In most cases, for most publications, this will be a losing battle. Still, I can’t help but feel that the moment is ripe for some modest progress here, and The New York Times might be the ones to do it.

I’m not the only one backing away from the social platforms, turned off by the chattering torrent therein. It seems harder every day to maintain a decent signal to noise ratio, which in theory is something the platforms themselves could change. They are developing better filters, and will continue to do so. But with their entire business strategies hinging on more eyeballs on more content for longer, they have a hard time actually making progress on this problem. Essentially, I want tools that make it easier for me to spend less time on Twitter and still find all that I want. That’s in conflict with Twitter’s business plan.

Of course, The Times wants eyeballs on its content for as long as possible, too. But the fact that its business model now includes a subscription component helps here. Once I’m paying for the content, the economics of providing a product I’m not obsessively checking constantly work better. So I’m hopeful that The Times might successfully offer a news feed app that works without being a hopelessly addictive time suck.

A big piece here is that NYT Now plans to include some stories from elsewhere, overcoming one of the biggest barrier to news apps in general, which is that no single publication can ever have all the content you want to read.

There’s also the price. My hesitation in paying for The Times is well documented, and can be summarized as:

1) As much as I like the Times, I don’t need it. And I’d rather be asked to support good journalism than forced to pay for it.

2) If I am going to pay money to support good journalism, I want to know that my money is going directly to that cause.

I wish The Times were structured more like The Guardian, with an endowment funding its efforts. But that’s not the case, and I’m more amenable to paying for it than I was a couple years ago, for various reasons. But at nearly $9/week, the price for the full digital subscription is still high for me.

My basic benchmark in terms of what feels reasonable is Netflix and Spotify: the $7-10/month range. Sure enough, NYT Now falls squarely in that range, at $2/week. That’s getting cheap enough that I might pay merely to support the paper’s mission.

The final reason I’m excited is that I’ve found Circa’s Android app more satisfying than I would have thought, in large part because it is sparing with its notifications. (It seems to only push out truly major news, as opposed to The Times, which pushes alerts about the Final Four.) The presentation in Circa is so clean and condensed that it for the first time has me inclined to see real value in the news app, above and beyond the content, where previously it has always seemed that an RSS reader or Twitter handles the app layer just fine.

For all these reasons, I could see NYT Now working for me (once it comes out for Android). It’s a relatively inexpensive way to support good journalism, and a less noisy way to stay on top of the news as opposed to social media. And they’ve finally learned the lesson that aggregation doesn’t dilute the brand. I may finally have found a news app I want to pay for.

(Note: here’s another Nieman review.)

Who cares if the stock market is rigged?

One of the most interesting bits of the debate over high frequency trading sparked by Michael Lewis’s new book is the question of why we should care that some Wall Street firms are ripping off other ones. One reason might be if Main Street’s money is disproportionately tied up in the funds that are getting ripped off. That raises the question of just who has how much money in the stock market. Not surprisingly, the vast majority of stock is owned by the wealthy.

Start with this overview of the stock market via Business Insider:

chart-of-the-day-who-owns-the-stock-market-november-2012The first thing to note here is how small the pension fund slice is, and how it has shrunk over time. So pensions make up a relatively small slice of stock ownership, but the average American still might be holding stock via a mutual fund or as individuals, right? Well, here’s a look at the percentage of Americans at each income level that own any stock or bonds, including via a mutual fund:

Screen Shot 2014-04-04 at 1.34.15 PM


And here’s a look at how much Americans have put away for retirement, by income level:

Screen Shot 2014-04-04 at 1.36.23 PM











As for the households piece, fewer and fewer Americans are investing in the stock market:

Screen Shot 2014-04-04 at 1.39.27 PM

So can we put this all together to get a sense of how much of the stock market is owned by whom? Here’s one attempt that goes beyond just stock by the Institute for Policy Studies via ThinkProgress:

Screen Shot 2014-04-04 at 1.40.48 PM


And here’s confirmation that that distribution holds for stock by The New York Times:

The richest 10 percent of households own about 90 percent of the stock, expanding both their net worth and their incomes when they cash out or receive dividends.

The point is clear: the vast majority of stock is owned by the rich, even once you take into account retirement funds. Of course, that doesn’t mean we don’t have to care about the stock market. Among other reasons, the small amount of money the non-rich have invested there represents a decent chunk of their wealth.

But it’s nonetheless worth keeping this distribution in mind when talking about who is ripping off whom in the stock market. For most Americans, the great economic injustice is stagnant wages, not high frequency trading.

UPDATE: Ben Walsh vs. Michael Lewis on stock ownership as it relates to HFT.



Eric Posner on Bitcoin as a payment protocol

Interviewed by Goldman Sachs, he comes to much the same conclusion as I did in my previous post on the attendant fees:

Eric Posner: I think there could be some advantages of usingBitcoin over existing payment systems, but these advantages are not as obvious as they might seem. For example, probably themost compelling advantage is that Bitcoin transactions seem to becheaper. Existing payment systems are often quite expensiveeither because somebody effectively has a monopoly, there are alot of government regulations that are costly to comply with, or thecompanies that offer these services provide certain protectionsthat people want and are willing to pay for.In the case of Bitcoin as it stands now, these costs are largely avoided, at least to the extent that you can technically send bitcoins from one wallet to another wallet without incurring fees;no middlemen are required to do this. The problem is that mostpeople will end up relying on intermediaries when they use bitcoin,not in least part due to security concerns around storing bitcoin onhard drives that can crash, be hacked, or, as in one famous case,thrown away. Most people will buy bitcoins from exchanges and use bitcoin service providers like Coinbase or Bitpay to store their bitcoins and transfer money to somebody in another part of the country or the world. Then that person will maintain their bitcoinswith a service provider and/or will convert the bitcoins back into themoney they use. And perhaps the same or other intermediaries willprovide insurance or protection from exchange rate volatility. Whenyou throw in all of these things, the effective price of using bitcoinis going to be greater than zero. Is it going to be as much as itcosts right now to use your credit card or a bank wire? Maybe not, but it is too soon to tell.

The economics of Bitcoin as payment protocol

As excitement over Bitcoin as a new form of currency has met with strong pushback from the economics world, the smarter commentators have shifted focus to cryptocurrency as a new way to move money around on the Internet. But how would that really work? The first thing you hear in these discussions is that Bitcoin transactions bypass traditional credit card fees. My first question was why? Specifically, isn’t at least part of the point of those fees to cover necessary services like fraud protection that will need to be implemented with Bitcoin as well? I asked Bitcoin entrepreneur Jeremy Allaire about this in an interview for HBR, and he gave a plausible answer. Basically, with Bitcoin no single entity bears the cost of clearing the transaction or maintaining the network to do so. Companies like Allaire’s Circle still need to provide anti-theft protection, and so it’s far from certain that the economics for Bitcoin will be superior, but for the purposes of this piece I’ll grant his assumption. As we’ll see, things only get more complicated from there.

So my starting point is the assumption that a Bitcoin transaction is cheaper than a (non-cash) dollar transaction. Could you build a payment app that just uses Bitcoin to exchange dollars, similar to how Business Insider’s Joe Weisenthal describes, and if so would it be cheaper? Here’s Weisenthal:

Bitcoin is fundamentally a way to make transactions in a fiat currency. If you want to sell me something for $850, I could pay you in cash, credit card, via PayPal, bank wire, or possibly Bitcoin. How many Bitcoins this transaction requires (currently it would be right around one) is a function of fluctuating Bitcoin prices, but essentially we’re carrying out a dollar-priced transaction and using the Bitcoin as the payment system.

You could build such an app, but you’d quickly run up against another kind of fee. If I want to buy $10 worth of something from a merchant with dollars, but using Bitcoin, the app needs to make two currency conversions. First, you need to take my $10 and convert it into Bitcoin, for which you’ll be charged a fee. And then you need to take the x Bitcoin that is transferred to the merchant and you need to convert it back into dollars. So you effectively have two transactions here, each with a fee that is roughly comparable to a credit card transaction. So you’re not really saving any money.

You might think that a way around this would be to use batching — as an example think of the way people put money onto Starbucks gift cards. I put on $50 which requires a one-time conversion fee to turn into Bitcoin, and then I make several purchases with Bitcoin over time. This does save you some money, since the Bitcoin transactions are cheaper than credit card transactions (per our assumption) and because converting all $50 into Bitcoin at once saves some money. (This is because interchange fees and currency fees both involve a mix of flat and percentage-based fees. So if I only do one fee-based transaction in the beginning and then do a bunch of cheap/free Bitcoin transactions, I’m saving a bit relative to the flat fee that would be assessed doing each transaction in dollars.)

But there’s a problem here, too. Why not just scrap Bitcoin, and batch payments in dollars? The savings in the above case are really coming from lumping together transactions to minimize flat fees — but we don’t need Bitcoin to do that. Indeed, the mobile payment startup LevelUp does this, best I can tell, on a monthly basis. (An email from the company says it’s to “to help local businesses save on card processing fees.”) Bitcoin does have the advantage of verifying at each transaction that the buyer actually has the money, so you don’t get to the end of a batch period and find out someone can’t pay. But the fact that LevelUp is using this approach suggests that the advantage might be quite small.

So we’re really back to square one, or at least close. Unless currency conversion fees are lower than interchange fees, I have trouble seeing how Bitcoin amounts to a cheaper way to move dollars around the Internet. In other words, if Bitcoin doesn’t work as a currency, I struggle to see how it will work as a payment protocol. Of course, we are early days and all that. Mostly, I’d really like to read a detailed account of how Bitcoin-as-payment-protocol would work so please point me in the right direction.


In the process of thinking all this through, I came up with an interesting hypothetical. Assume for a minute that the fixed supply of Bitcoin suggests its value will appreciate over time (and put aside volatility for a moment). Imagine you built a digital payment app that used Bitcoin but didn’t bill itself as being about Bitcoin at all.

Here’s how it would work: I put $20 onto the app, which behind the scenes is immediately exchanged for Bitcoin. But rather than telling me I have x Bitcoin to spend, you simply guarantee I can spend $20 dollars. I spend that money over time — say a month — during which the value of Bitcoin appreciates. Because of that, once I’ve spent my $20 there’s still some Bitcoin left over. And because this isn’t a Bitcoin app — I don’t know/care that that’s how you’re processing my payments — you just go ahead and pocket that residual to cover fees and for a profit.

Of course, the company behind the app needs to worry about the loss it would face if the price of Bitcoin goes down over that period. But if you believe there’s a fundamental deflationary bias in Bitcoin, that suggests that to the extent that it is used, its value will appreciate. Moreover, it may be advantageous to have firms rather than individuals taking on the risk around Bitcoin’s price volatility, in both directions. I don’t have to worry (much) since my dollars are guaranteed so long as the app company is solvent.

What really interests me in this scenario is the fact that the deflationary spiral that supposedly occurs when a currency appreciates over time is driven by hoarding. If you know your money will be worth lots more tomorrow, you’re unlikely to spend it. But in this case, I don’t really even know that I’m using Bitcoin so I’m unlikely to hoard it. So demand keeps ticking along. Could such a dynamic help mute the recessionary tendencies of a fixed supply currency?

No doubt there is a reason why this whole thing wouldn’t work — look forward to hearing folks’ thoughts as to why not.

The real danger of Bitcoin

It’s been interesting to see the debate over Bitcoin start with currency, move on to payments, and now there are rumblings that its real use isn’t for either. Rather, cryptocurrency could be a way of marking digital goods as unique, making them in effect rival. Technology Review explains:

Or take digital art. Larry Smith, a partner at the business architecture consultancy The matix and an analyst with long experience in digital advertising and digital finance, asks us to “imagine digital items that can’t be reproduced.” If we attached a coin identifier to a digital image, Smith says, “we could now call that a unique, one-of-a-kind digital entity.” Media on the Internet—where unlimited copying and sharing has become a scourge to rights holders—would suddenly be provably unique, permanently identified, and attached to an unambiguous monetary value.

This would almost certainly be bad news for the Internet. The ability of users to copy digital content has put necessary pressure on rights holders to lower costs, change business models, and innovate. We live under a ludicrous intellectual property regime in which rights holders lobby their way to ever-extending copyright terms. Making it easier for rights holders to enforce copyright online would result in more expensive content and would take all the pressure off both for improved digital distribution models and intellectual property reform. Put simply, the non-rivalry of digital goods is one of the things that has made the Internet such a boon for consumers. If cryptocurrency undoes that, it would be a shame.

How will self-driving cars solve the trolley problem?

Recall the classic utilitarian morality puzzle (via Wikipedia):

There is a runaway trolley barrelling down the railway tracks. Ahead, on the tracks, there are five people tied up and unable to move. The trolley is headed straight for them. You are standing some distance off in the train yard, next to a lever. If you pull this lever, the trolley will switch to a different set of tracks. Unfortunately, you notice that there is one person on the side track. You do not have the ability to operate the lever in a way that would cause the trolley to derail without loss of life (for example, holding the lever in an intermediate position so that the trolley goes between the two sets of tracks, or pulling the lever after the front wheels pass the switch, but before the rear wheels do). You have two options: (1) Do nothing, and the trolley kills the five people on the main track. (2) Pull the lever, diverting the trolley onto the side track where it will kill one person. Which is the correct choice?

How should we program robots to answer this question? Specifically, what about self-driving cars? Should they be programmed to injure or kill their driver in order to save many others? The question is raised at minute three of this short video on robots and ethics. The whole video is worth your time.