A Twitter Experiment

I’m a long-time Twitter skeptic.  It’s difficult for an economist to see a 140 char lmt as a ftr.  My journalist friends tell me I’m dead wrong.  And a recent long and boozy evening with co-founders Evan Williams and Jason Goldman convinced me to give it a try.  Is Twitter worth the hype?  Let’s find out.

Today I’m beginning my Twitter Experiment. I’m now tweeting @justinwolfers. I’m going to keep this up for a couple of weeks as a “burn in” period—basically so that I can learn the ecosystem before my experiment begins. Then on the morning of August 1, I’m going to wake up, and flip a coin. Heads, I’ll open Twitter; tails I won’t.  And I’ll do the same on August 2, and then every day for three months. If the coin comes up heads, it doesn’t necessarily mean that I’ll tweet, just that it will be a Twitter-aware day; I’ll consume the stream, and tweet away if I feel the need. Tails, and I’ll simply tweet “Tails, goodbye,” close the stream (unless I need it for research) and then resist the urge to tweet for the rest of the day.

So far, designing the experiment is easy. But what are the right outcome variables to track? I can think of three things that Twitter does:
1.      Distracts me, sucking valuable time from serious research.

2.      Broadcasts me, perhaps increasing my influence.

3.      Informs me, which may make me a better economist.

Here’s my question for you: How can I measure the influence of my Twitter Experiment on each of these outcomes? I think measuring distraction is easiest—I’ll just evaluate how productive I was each day, using a subjective 0-10 scale. I have some ideas about measuring influence, but none are particularly compelling. And I’m baffled by how I might measure the extent to which Twitter is informing me, making me a better economist.

Crowdsourcing can surely improve my experiment. Do you have any ideas for how better to measure distraction? Or ideas for useful indicators of influence or impact that I can track?  And how best to measure how informed I am? And does anyone else want to join me in this experiment? More data = more insight.

Please add your ideas in the comments.  Or if your idea is ez 2 expln, just tweet me.  #twitexpt.


Lee

"Tails, and I’ll simple tweet “Tails, goodbye,” close the stream (unless I need it for research) and then resist the urge to tweet for the rest of the day."

Unless you need it for research. I think you just demonstrated some value right there.

BSK

The experiment is already undermined if you are going on it on the days you are not supposed to. If you're off Twitter, you should be entirely off Twitter. Otherwise, the whole thing is bunk.

KevinH

for 1, there are actually timers available for the computer that will help you track and objective number of hours spend on twitter. I think the subjective scale would also be useful, but it would also be interesting to both the distribution of time spent with twitter and if the time spend on a daily basis correlated with any other factors.

for 2, can you track web hits to blog posts by you? It wouldn't measure professional influence, but it would be a decent variable to track.

Likewise for 3, you could count the number of articles you read each day with a little tally sheet. I'd say you could do 2 tallies, one for academic articles and one for non-academic articles.

Michael Vukovich

Influence: how many people follow you and retweet what you say...a good measure for how much you are spreading your word to the masses. Also check out #, if you're hashmarked often it means people are tagging you in a lot of posts.

Informing: possibly keep track of how often you 'use' information you discover on twitter? Subjective I know however you're already using a subjective scale for productivity

raulbajob

Justin, I cannot see why you are going to flip a coin?¿?
If you just want to measure distraction, use a stopwatch! And a a possible way to measure your influence is, for example, 1 point for each new follower in a day, 2 points for each metion, and 3 points for each retweet. At the end of each day, just sum the points up.

Sol

I think your experiment is very interesting, I am also skeptic about twitter.
I'd measure the influence considering the number of followers. I'd study how your tweets affect the number of followers, whether your tweets increase or decrease it. I'd even intentionally tweet both "interesting" and "stupid" comments, that is to know if being boring and uninteresting makes people stop following me.
And of course I would also consider how many replies you get for every tweet. If you can make someone reply you about anything, then I guess you can assume you had some influence on him (which is just what has happened to me, replying at this moment).
Hope this helps!

AZ

I see two problems with recording productivity. As already mentioned, it's highly subjective (and potentially affected by your knowledge of which kind of day it was). But I would also think that productivity is so variable, and so subject to other influences, that it would be unlikely that an experiment with a relatively small N (even 92) would reveal much.

Kathleen Lisson

'Distract' and 'Inform' variables both depend on which twitter feeds you are following.

It may be interesting to measure the level of feedback in an hour that you get on a question posted both on this blog and on twitter. With twitter, your question could be retweeted, giving you potentially increased feedback from a wider community not usually exposed to this blog.

Kaia

Do you have any ideas for how better to measure distraction?
: measure how many times twitter makes you to think about it during the day or limited time such as time for experiment or research. For example, you could think you'd like to check twitter three times for two hours research time. This means that you were bothered by twitter three times. (This sounds too subjective, tho. but I think 'distraction' is very sujective concept) As for me, twitter is bothering me almost all day long; I think I'm addicted to it :(

Or ideas for useful indicators of influence or impact that I can track?
: Use retweet function. People retweet other's mentions when they are useful, educational, or just fun. And you are easily able to track not only which one was followed but also how many times the mention was followed. 

Joshua Northey

What I don't understand about Twitter is that of the three things you mentioned, I don't see how it is better at them than its alternatives in any way.

It is great for journalists, and I can see how that would help it explode in popularity because journalists are influential, but there is no there "there". It has no value for your average person.

If you need to find out what is happening RIGHT this instant, and cannot wait 4 hours before it its all the major news sites Twitter is for you. Other then I am at a loss.

PHB

See the Scott Adams blog entry for today as a start. Measure in someway your creativity on both days. Create a scale of how easy or not ideas flow. All else has too many independent variables.

Ben

There are a bunch of (imperfect) free tools that will measure your Twitter influence for you - Klout, Peer Index, Retweet Rank, Twitalyzer

Terry

you need to follow more celebrities. that is where the fun is.

Shane

I'm still not sure, but I wonder if 140 characters is only enough to confirm someone's prejudice by parroting a hollow slogan, but not enough to challenge prejudice. I also joined, but use it rarely. I find it a frustrating mode of communication.

Encelade

When I read something interesting, I, usually, re-write it somewhere, save it, download it, whatever-you-want it. But I make sure the piece of information will be somewhere safe for me to use later.
I suppose I'm not the only one to do that.
Just count how many tweets a day were worth saving for later. And compare with the number of bits of information you get on your Tails days.

Cordialement

Anon

I think the outcome of this experiment will depend on who you choose to follow, and why.

If I wanted to ensure I wasn't distracted, I'd simply not follow any of those accounts that serve mostly for entertainment purposes (ie. comedians, celebrities, friends and others who tweet for social purposes only, etc). However, if I wanted to focus on a particular subject and chose to only follow accounts that tweet relevant information, then I believe chances of distraction will go down significantly.

I think for those who perhaps do not have many friends who use Twitter, personal usefulness is minimal. However, for those who have a good chunk of friends and acquaintances who use Twitter regularly, think of it as a huge conversation that you can either read or choose to participate in if you have a view to express.

Ben

If the 140 character limit bugs you, why not try out Google+?

I get the feeling that it might someday serve as more of a Twitter substitute than a Facebook substitute.

Jen

Influence: Retweets and mentions, maybe also @'s.

Being informed: Number of valuable posts / articles etc. you were directed to via twitter. Naturally, just like the distraction variable via a "productiveness"-scale, this would be subjective, but since it's meant for a personal assessment, why not?

financial-translator (@miguelllorens)

Regarding one way to measure point number 2 (whether it broadcasts you), ask the person who runs the Freaknomics blog to compare traffic to your blog posts from Twitter compared to your pre-Twitter numbers or, better yet, the traffic on non-Twitter days.

@4g5

You didn't take one thing into account:

If you tweet someone, say someone of influence, or someone without using your economical knowledge, you could perhaps make a difference in their decision making process.

Twitter is all about expanding the pipeline from fat cats to the users (people, firms, etc). So by becoming a user, you theoretically have the potential to alter the behavior of the fat cats, as you now have a voice.

Under the current scope of your experiment, you're not going to do very well at all, because you're looking at it from the wrong lens, "What do I gain?" The internet, and Twitter for that matter, are all about contributing to a larger collective.

You need to look at it more so in what you have to offer, rather than what you personally can gain- how scientific is that?

StatSpotting

Did you know, The World Tweets Once For Every 1778 E-Mails Sent?

http://statspotting.com/2011/03/the-world-tweets-once-for-every-1778-e-mails-sent/

Eben Lazarus

How about adding to the experimental design by having someone you work with (another economist or a research assistant) assess your daily productivity instead of you doing it yourself? You would need to keep a daily list of things accomplished, which you would report to this colleague at the end of every day without telling him/her whether it was a Twitter or non-Twitter day. Then he/she would grade your productivity on your 1 to 10 scale. This would add an element of blindness to the experiment.