Stop Twitter Spam

Twitter’s scale problems stem from the fact that the workload increases geometrically as every message people send is forwarded to each follower.  The more followers people have, the slower twitter is.  The follow-spammers just increase that load.  Some of these spammers follow 25,000 people.  That’s 25,000 extra message each time any of those 25,000 tweet.  There are hundreds of follow-spammers, so that’s hundreds of thousands, maybe millions, of extra tweets.

- /dev/null/kevin: Twitter Needs to Delete the Spammers

Twitter provided another update on their developer blog today. In this latest update, Twitter answered questions about their recent stability issues. One of the questions asked if there is anything that Twitter users can do to lighten the load.

Twitter’s response:

The events that hit our system the hardest are generally when “popular” users – that is, users with large numbers of followers and people they’re following – perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that’s behavior we have to limit on our side.

So who are the people who are running scripts to follow thousands of people at a time? I’m willing to bet that most of these people are spammers or over-zealous marketers who are trying to follow as many people as possible in a short amount of time. It’s not really clear from the blog post what Twitter is doing to limit these scripts but it’s good news that they are taking steps to limit this type of behavior.

Looking back on this, it looks like one of the key features that triggered Twitter’s massive growth over the past year – their open API – has also bitten them in the ass because it enabled things like the mass-follow scripts, which have caused stability problems. This left their platform wide open for abuse by “black hat” marketers who have been treating the Twitter user base like a free mass mailing list. And the Twitter community paid the price with frequent downtime. It’s pretty clear that whatever throttles and checkpoints Twitter had in place to limit their exposure to rogue applications was not enough. They learned this lesson the hard way.

This is a critical time for Twitter. Many users are moving over to other tools like FriendFeed, Jaiku, and Pownce while Twitter sorts out their technical problems. Is this a temporary move or are these users gone for good? I’m really hoping that this turns out to be a temporary glitch and not a repeat of what Friendster went through a few years ago.

Update:

Below is a screenshot of a tool that is designed to follow massive amounts of Twitter users. There are other many other tools similar to this one.

Today Robert Scoble blogged about some new proposals for charging a premium for people who follow lots of people on Twitter. He rejects the idea and disputes the claim that users such as himself, Lea Laporte, or Jason Calacanis put more “strain” on Twitter’s system.

I’m not even going to attempt to get into that discussion because none of us really know exactly how screwed up the situation is with Twitter’s architecture. Based on blog posts like this from Twitter’s developers we know that they have work to do to turn Twitter into more of a true messaging system.

I actually like the idea of Twitter charging a premium for people who follow a large number of people because this would be a huge spam deterrent. Right now spammers are having a free lunch and Twitter (and indirectly, the Twitter community) is picking up the bill. There is absolutely no barrier whatsoever to following as many people as they they’d like in an attempt to get people to click through to their web site, either from their profile page or from Tinyurl’s that lead to a splog or a web site loaded with affiliate links.

The fact that most people are ignoring or blocking these spam users isn’t the issue. The issue is Twitter’s system resources getting chewed up by the mass-following activity (using bots scraping the public timeline or cloning following lists from popular Twitterers like yourself), not to mention all of the blocking activity that results from people getting rid of these annoying spammers. I wouldn’t be surprised if some of this mass-follow activity was the culprit or at least contributed to some of the recent Twitter outages.

Charging a premium would be a huge deterrent against the spammy mass-follow behavior. I might be wrong on this, but I think very few spammers would continue trying to squeeze click-through’s from Twitter if they had to pay for the privilege of following thousands of people.

I want to make it very clear that I’m not calling Robert Scoble a spammer (or Jason or anyone else who follows thousands and genuinely interacts with their followers) because he is following lots of people. I’m referring to the bots that start following 10K people in one day and run on auto-pilot (auto-posting tweets). With that said, I do think that these guys are getting a massive amount of publicity out of Twitter and, as a result, should be supportive of the pay model. I know that Jason is supportive of this as long as it guarantees him better uptime. Robert, would you reconsider your position on this if this resulted in less spam on the system?

Update:
I want to clarify that I think people should be charged for following lots of people, not for being followed by lots of people. So under this model, someone like Leo Laporte, who is followed by over 37,000 people but is only following 437 would not pay as much as Robert Scoble, who is followed by fewer people (25,284) but is following over 21,000.

Stephen Tann has an interesting proposal for reducing spam on Twitter.  Charge extra if you follow more than $200 people.

All accounts can post unlimited tweets for free. Twitter needs content to create value for the user base (hello Livejournal, remember that? Free accounts generate the content that make the paid accounts valuable).

Free and subscriber accounts get a base of 200 follows. Users can purchase an additional 200 Follow credits for a flat fee of $5/month for 200, or $50/200 for 12 month package. This could dramatically cut the auto-follow spam accounts out of the system (or compensate twitter for their drain on service). A power user like Robert Scoble would be up for $105 a month. If Scoble is getting $105 or more value from following 21,145, then it’s a cheap deal. If not, then it’s good time for Robert to cutdown on the number of people he follows.

Business rationale: Based on a remark from the series of posts on scaling microblogs series at Hueniverse,

Going through a timeline request, the server first looks up the list of people the user is following. Then for each person, checks if their timeline is public or private, and if private, if the requesting user has the rights to view it. If the user has rights, the last few messages are retrieved, and the same is repeated for each person being followed.

Lower the number of follows per user to a level that the user self moderates as valuable, and you’ll lower the number of calls to the timeline. Given free accounts can have at least 45000+ follows (that’s the highest score on a spambot account I’ve personally), that’s a lot of database work. Charge for the use of follows and the abuse of the system should decline rapidly.

Reaf the complete post here.

An Apple a Day: Top Ten Rants on Twitter

No surprise – Spam makes the list at #9

Andrew Kinnear hits the nail on the head with his take on how to use Twitter as a marketing channel. Let your customers do the talking…

If I’ve got a product or service and I want to spread the word virally, then all I really need is a hard-core group of influencers– a seeding group. If it’s the right message, to the right target audience, then the right influencers can spread the word faster than a million poorly targeted emails.

- Andrew Kinnear: Is Twitter a real marketing tool or a time waster? (5/13/08)

Twitter Product Manager Jason Goldman provided an update today on Twitter’s plans for fixing the Track Spam issue.  Here’s part of that update:

1) Track is a feature that allows you to receive (via sms or im) all Twitter updates matching certain search terms. A common use is to track one’s own name. This is useful to find replies directed at you from people you don’t follow.

More info is here : http://help.twitter.com/index.php?pg=…

2) Currently, this feature does not respect the block list. That means that if you block a given user and they create an update with a phrase you track, it will still be delivered to you.

3) We consider this a bug not a feature. It is our intention to change the track behavior such that it respects the block list. I don’t have a specific timeline for implementation; it’s not an easy change, technically. But if we could will it to be so, track would respect the block list.

TechCrunch has a post today about Twitter Spam titled Twitter Starts Blacklisting Spammers. I actually think the post is going to confuse some people. It mentions The Twitter Blacklist, which is not affiliated with Twitter, and a recent post on Stay N Alive, which covers a recent discussion on the Twitter Dev discussion group about how to detect users that have been “blacklisted” by Twitter. These are two completely separate things.

If you quickly scan the TechCrunch article you might come away with the impression that TechCrunch is saying that the Twitter Blacklist is what is being used by Twitter to identify spammers. If you read the article carefully you’ll see that they didn’t really say that. But at least one person came away with that impression and I’m sure there are many others.

From what I can tell, this is the process that Twitter is using to identify and deal with spammers:

1) Suspected spammers are reported to Twitter from this Help page. Another way that Twitter can be notified about a suspected spammer is a high volume of Block activity against a specific account.

2) Someone from Twitter reviews the account to determine if the Terms of Service have been violated. It’s important to note here that they are mainly looking for people who are contributing to “Follow Spam”. This is where someone follows thousands of people at a time, resulting in a New Follower email. From what I can tell, all of these spam requests are reviewed by an actual person and they are not just using a simple formula like the Following:Followers ration to make their decision.

3) Once Twitter has determined that it’s a spammer they take action against the account. I have received a number of emails from Twitter about spam reports that I have filed. Click here for an example. Some typical actions that they will take are: removing their tweets from the public timeline, suspending update capabilities, and suspending follow/unfollow privileges. Based on the Stay N Alive it appears that Twitter is going to take a stronger actions moving forward and terminate the account versus just cripple it.

This type of coverage about Twitter Spam usually generates a lot of discussion. Here are some posts from people who question whether or not this is an issue:

• Daggle – Twitter Spam: Myth or Reality?
• I’m Not Actually A Geek – Spammers on Twitter and FriendFeed: Really A Problem?

Check out TechMeme for more discussion on this.  And of course, there has been lots of discussion on Twitter about this.

ProBlogger: 5 Tips to Grow Your Twitter Presence
Some good tips for getting more people to follow you without resorting to automated “Friend Adder” tools.