A Twitter Experiment

I’m a long-time Twitter skeptic.  It’s difficult for an economist to see a 140 char lmt as a ftr.  My journalist friends tell me I’m dead wrong.  And a recent long and boozy evening with co-founders Evan Williams and Jason Goldman convinced me to give it a try.  Is Twitter worth the hype?  Let’s find out.

Today I’m beginning my Twitter Experiment. I’m now tweeting @justinwolfers. I’m going to keep this up for a couple of weeks as a “burn in” period—basically so that I can learn the ecosystem before my experiment begins. Then on the morning of August 1, I’m going to wake up, and flip a coin. Heads, I’ll open Twitter; tails I won’t.  And I’ll do the same on August 2, and then every day for three months. If the coin comes up heads, it doesn’t necessarily mean that I’ll tweet, just that it will be a Twitter-aware day; I’ll consume the stream, and tweet away if I feel the need. Tails, and I’ll simply tweet “Tails, goodbye,” close the stream (unless I need it for research) and then resist the urge to tweet for the rest of the day.

So far, designing the experiment is easy. But what are the right outcome variables to track? I can think of three things that Twitter does:
1.      Distracts me, sucking valuable time from serious research.

2.      Broadcasts me, perhaps increasing my influence.

3.      Informs me, which may make me a better economist.

Here’s my question for you: How can I measure the influence of my Twitter Experiment on each of these outcomes? I think measuring distraction is easiest—I’ll just evaluate how productive I was each day, using a subjective 0-10 scale. I have some ideas about measuring influence, but none are particularly compelling. And I’m baffled by how I might measure the extent to which Twitter is informing me, making me a better economist.

Crowdsourcing can surely improve my experiment. Do you have any ideas for how better to measure distraction? Or ideas for useful indicators of influence or impact that I can track?  And how best to measure how informed I am? And does anyone else want to join me in this experiment? More data = more insight.

Please add your ideas in the comments.  Or if your idea is ez 2 expln, just tweet me.  #twitexpt.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.



  1. Lee says:

    “Tails, and I’ll simple tweet “Tails, goodbye,” close the stream (unless I need it for research) and then resist the urge to tweet for the rest of the day.”

    Unless you need it for research. I think you just demonstrated some value right there.

    Well-loved. Like or Dislike: Thumb up 15 Thumb down 1
    • BSK says:

      The experiment is already undermined if you are going on it on the days you are not supposed to. If you’re off Twitter, you should be entirely off Twitter. Otherwise, the whole thing is bunk.

      Well-loved. Like or Dislike: Thumb up 14 Thumb down 0
  2. KevinH says:

    for 1, there are actually timers available for the computer that will help you track and objective number of hours spend on twitter. I think the subjective scale would also be useful, but it would also be interesting to both the distribution of time spent with twitter and if the time spend on a daily basis correlated with any other factors.

    for 2, can you track web hits to blog posts by you? It wouldn’t measure professional influence, but it would be a decent variable to track.

    Likewise for 3, you could count the number of articles you read each day with a little tally sheet. I’d say you could do 2 tallies, one for academic articles and one for non-academic articles.

    Thumb up 2 Thumb down 0
  3. Michael Vukovich says:

    Influence: how many people follow you and retweet what you say…a good measure for how much you are spreading your word to the masses. Also check out #, if you’re hashmarked often it means people are tagging you in a lot of posts.

    Informing: possibly keep track of how often you ‘use’ information you discover on twitter? Subjective I know however you’re already using a subjective scale for productivity

    Thumb up 0 Thumb down 0
  4. raulbajob says:

    Justin, I cannot see why you are going to flip a coin?¿?
    If you just want to measure distraction, use a stopwatch! And a a possible way to measure your influence is, for example, 1 point for each new follower in a day, 2 points for each metion, and 3 points for each retweet. At the end of each day, just sum the points up.

    Thumb up 6 Thumb down 2
  5. Sol says:

    I think your experiment is very interesting, I am also skeptic about twitter.
    I’d measure the influence considering the number of followers. I’d study how your tweets affect the number of followers, whether your tweets increase or decrease it. I’d even intentionally tweet both “interesting” and “stupid” comments, that is to know if being boring and uninteresting makes people stop following me.
    And of course I would also consider how many replies you get for every tweet. If you can make someone reply you about anything, then I guess you can assume you had some influence on him (which is just what has happened to me, replying at this moment).
    Hope this helps!

    Thumb up 1 Thumb down 0
  6. AZ says:

    I see two problems with recording productivity. As already mentioned, it’s highly subjective (and potentially affected by your knowledge of which kind of day it was). But I would also think that productivity is so variable, and so subject to other influences, that it would be unlikely that an experiment with a relatively small N (even 92) would reveal much.

    Thumb up 4 Thumb down 0
  7. Kathleen Lisson says:

    ‘Distract’ and ‘Inform’ variables both depend on which twitter feeds you are following.

    It may be interesting to measure the level of feedback in an hour that you get on a question posted both on this blog and on twitter. With twitter, your question could be retweeted, giving you potentially increased feedback from a wider community not usually exposed to this blog.

    Thumb up 1 Thumb down 1
  8. Kaia says:

    Do you have any ideas for how better to measure distraction?
    : measure how many times twitter makes you to think about it during the day or limited time such as time for experiment or research. For example, you could think you’d like to check twitter three times for two hours research time. This means that you were bothered by twitter three times. (This sounds too subjective, tho. but I think ‘distraction’ is very sujective concept) As for me, twitter is bothering me almost all day long; I think I’m addicted to it :(

    Or ideas for useful indicators of influence or impact that I can track?
    : Use retweet function. People retweet other’s mentions when they are useful, educational, or just fun. And you are easily able to track not only which one was followed but also how many times the mention was followed. 

    Thumb up 0 Thumb down 0
  9. Joshua Northey says:

    What I don’t understand about Twitter is that of the three things you mentioned, I don’t see how it is better at them than its alternatives in any way.

    It is great for journalists, and I can see how that would help it explode in popularity because journalists are influential, but there is no there “there”. It has no value for your average person.

    If you need to find out what is happening RIGHT this instant, and cannot wait 4 hours before it its all the major news sites Twitter is for you. Other then I am at a loss.

    Thumb up 2 Thumb down 0
  10. PHB says:

    See the Scott Adams blog entry for today as a start. Measure in someway your creativity on both days. Create a scale of how easy or not ideas flow. All else has too many independent variables.

    Thumb up 0 Thumb down 1
  11. Ben says:

    There are a bunch of (imperfect) free tools that will measure your Twitter influence for you – Klout, Peer Index, Retweet Rank, Twitalyzer

    Thumb up 1 Thumb down 0
  12. Terry says:

    you need to follow more celebrities. that is where the fun is.

    Thumb up 0 Thumb down 1
  13. Shane says:

    I’m still not sure, but I wonder if 140 characters is only enough to confirm someone’s prejudice by parroting a hollow slogan, but not enough to challenge prejudice. I also joined, but use it rarely. I find it a frustrating mode of communication.

    Thumb up 0 Thumb down 0
  14. Encelade says:

    When I read something interesting, I, usually, re-write it somewhere, save it, download it, whatever-you-want it. But I make sure the piece of information will be somewhere safe for me to use later.
    I suppose I’m not the only one to do that.
    Just count how many tweets a day were worth saving for later. And compare with the number of bits of information you get on your Tails days.


    Thumb up 1 Thumb down 0
    • Anon says:

      I think the outcome of this experiment will depend on who you choose to follow, and why.

      If I wanted to ensure I wasn’t distracted, I’d simply not follow any of those accounts that serve mostly for entertainment purposes (ie. comedians, celebrities, friends and others who tweet for social purposes only, etc). However, if I wanted to focus on a particular subject and chose to only follow accounts that tweet relevant information, then I believe chances of distraction will go down significantly.

      I think for those who perhaps do not have many friends who use Twitter, personal usefulness is minimal. However, for those who have a good chunk of friends and acquaintances who use Twitter regularly, think of it as a huge conversation that you can either read or choose to participate in if you have a view to express.

      Thumb up 0 Thumb down 0
  15. Ben says:

    If the 140 character limit bugs you, why not try out Google+?

    I get the feeling that it might someday serve as more of a Twitter substitute than a Facebook substitute.

    Thumb up 4 Thumb down 1
  16. Jen says:

    Influence: Retweets and mentions, maybe also @’s.

    Being informed: Number of valuable posts / articles etc. you were directed to via twitter. Naturally, just like the distraction variable via a “productiveness”-scale, this would be subjective, but since it’s meant for a personal assessment, why not?

    Thumb up 0 Thumb down 0
  17. financial-translator (@miguelllorens) says:

    Regarding one way to measure point number 2 (whether it broadcasts you), ask the person who runs the Freaknomics blog to compare traffic to your blog posts from Twitter compared to your pre-Twitter numbers or, better yet, the traffic on non-Twitter days.

    Thumb up 0 Thumb down 0
  18. @4g5 says:

    You didn’t take one thing into account:

    If you tweet someone, say someone of influence, or someone without using your economical knowledge, you could perhaps make a difference in their decision making process.

    Twitter is all about expanding the pipeline from fat cats to the users (people, firms, etc). So by becoming a user, you theoretically have the potential to alter the behavior of the fat cats, as you now have a voice.

    Under the current scope of your experiment, you’re not going to do very well at all, because you’re looking at it from the wrong lens, “What do I gain?” The internet, and Twitter for that matter, are all about contributing to a larger collective.

    You need to look at it more so in what you have to offer, rather than what you personally can gain- how scientific is that?

    Thumb up 0 Thumb down 0
  19. StatSpotting says:

    Did you know, The World Tweets Once For Every 1778 E-Mails Sent?


    Thumb up 0 Thumb down 0
  20. Eben Lazarus says:

    How about adding to the experimental design by having someone you work with (another economist or a research assistant) assess your daily productivity instead of you doing it yourself? You would need to keep a daily list of things accomplished, which you would report to this colleague at the end of every day without telling him/her whether it was a Twitter or non-Twitter day. Then he/she would grade your productivity on your 1 to 10 scale. This would add an element of blindness to the experiment.

    Thumb up 0 Thumb down 0
  21. The Regular Joe says:

    First of all, welcome and good luck
    I have a comment about your #1 variable – productivity
    It is inaccurate to measure it so simply
    I think you will find being “twitter aware” keeps you on edge writing wise and more tuned to everything else.
    In the mean time I offer myself to participate as a content provider that may or may not affect you

    Thumb up 0 Thumb down 0
  22. Nathan says:

    I think you have too many subjective variables for such a small sample of people. To measure how twitter distracts or informs you can be influenced by:

    * Personal feeling towards Twitter as a social media on that given day
    * Who you follow
    * Who follows you
    * How your interests correlate with those of people you follow/who follow you
    * Mood and productivity ignoring the possible influence of Twitter (would I just have found something else to distract me today?)

    Points 1 & 5 are easily reconcilable by adding them as data points but 2, 3 & 4 can only be accounted for by widening the sample, in my opinion. Perhaps you could measure distraction by comparing time spent on Twitter to how productive you felt off of Twitter (you have already conceded you’ll look at Twitter even when you get a tails – why bother flipping the coin at all?). Similarly you could perhaps tally how many ideas Twitter gave you in a given day. Yet neither of these can take into account who you follow and who follows you.

    Similarly measuring how well you are broadcast creates issues. How will you measure this? The most common way is to see how often you are retweeted but, as mentioned before, that depends on your followers being the right people. If you are tweeting about economics and yet have very few economists following you, you may either have a massive amount of retweets or none at all.

    You have already tried to create a control group with the tossing of a coin. Perhaps what you really need is to use your twitter account to encourage more people to join in the experiment. That way the number of participants would help you judge how well you are broadcasting (keeping people interested in the experiment). Also if you created a questionairre asking people about Twitter habits, followers, the people they follow etc., it might help you discover *HOW* to make Twitter less distracting and more informative by comparing this to their results.

    Thumb up 1 Thumb down 0
  23. charlie1 says:

    You should be able to measure the volume of traffic (visits) to your blogs/articles coming from twitter and compare to your pre-twitter time. Google anaytics is great for this. Should be a good guage of increasing influence

    Thumb up 0 Thumb down 0
  24. achilles3 says:

    why am I rooting for you not to become a Twitter guy…
    If that means you post less here than that is a significant drawback for me

    Thumb up 0 Thumb down 0
  25. Jeffrey K says:

    What about tracking your “addiction level ” over the life of the experiment and also see how much the non twitter days bother you. Does that change over time?

    Thumb up 0 Thumb down 0
  26. Marissa says:

    To measure how distracted you are on your “heads” days, you may want to tweet the moment you distract yourself with Twitter, and then tweet again when you are done (perhaps tweeting the words “begin” and “end,” respectively). That way you have an indicator of when you started and stopped and allow yourself to get lost in distraction (as one typically would online) without feeling like you are racing against the clock with a stopwatch.

    I agree that measuring your crowdsourcing with retweets/hashtags/followers would be useful. Perhaps one point for followers, two points for retweets and hastags (since that means you are being actively recognized).

    Measuring how informed you are may be more difficult. Perhaps listing everything you learned from Twitter at the end of the day, like a mini memory quiz?

    Thumb up 0 Thumb down 0
  27. Tim says:

    For item 2, something like http://topsy.com/ could help. For three even favouriting links you liked or informed would begin to build a list you could cross-compare with your existing sources.

    As for item 1, that would seem to be lack of result from items 2 or 3, or a lack of self discipline…

    Thumb up 0 Thumb down 0
  28. @mtbmobile says:

    RE: how best to measure how informed Iam? Evaluate the relevancy of everything your read via your ‘old school’ research methods, sources, tactics… and then do the same using twitter as a launch pad into the most relevant, filtered, real-time data on any concurrent event worldwide. Sign-up for a good post processing tool (google it, there are many) for historical tweets that rank each tweet base on based upon relevance, user klout, and other analytic measures.

    Also, I agree with the first comment. If it’s tails, no peeking at Twitter (regardless if you need it for research).

    Thumb up 0 Thumb down 0
  29. Keren says:

    Influence: I found this blog today because of a tweet and chose to Follow it on twitter – and then read 4 or 5 more blog postings before I found this one… of course what I found was the freakonomics handle, not yours. But still –

    Thumb up 0 Thumb down 0
  30. Jw says:

    I am all for measuring things (…) but I feel we are often measuring too much. What not such try it out for a while a keep books on the benefits (new insights, information you would not have found otherwise, anecdotal evidence) and the costs (mostly time). After six weeks, check for a positive balance.

    Here is a different way to look at Twitter: the chance to share a thougth a day with the world. Don’t expect anything in return. Honestly, I don’t get the coin thing. Then again, I am not an economist!
    (self promotion: some blogs I wrote about using twitter http://bit.ly/cys9K7 http://bit.ly/dpBpzB http://bit.ly/i8NdCD )

    Thumb up 0 Thumb down 0
  31. rer says:

    Well, it may be more useful/efficient if you can link your Twitter account to one you are already using. I know this is possible with Facebook, and I’ve seen some indirect ways to do this with Google+ (I hope that Google will implement a full fledged connection at some point). That way, with your accounts linked, when you post to Facebook, it automatically posts it to Twitter.

    Thumb up 0 Thumb down 0
  32. Mike N. says:

    Just thinking about it quickly, I think on days you check the twitter feeds, you will find yourself less productive, mainly cos you are stopping to check the latest ‘tweet’ (or whatever the damn things are called), this will stop you doing what you are doing, and waste seconds, as you put your pen down, find your blackberry, open twitter, read it, process the infomation, then put down etc. Kind of the same way that having your boss ring you for twenty minute updates means you get no work done.
    Just my tuppence worth from the UK.

    Thumb up 0 Thumb down 0