Home Products Plans & Pricing Help Blog About

TweetReach Blog

Thoughts on social media analytics from the makers of TweetReach

Archive for the ‘firehose’ tag

TweetReach: Where our Twitter data comes from

without comments

We use Gnip so we don’t have to send poor Merle down into the Twitter data mines.

We’ve written before about the five questions you should be asking your social analytics provider, and we wanted to make it clear what you’re getting when you choose TweetReach Pro. If you still have questions after you read this, feel free to share them in the comments below, or drop us a line. We’ll be happy to help answer them!

Does Union Metrics have access to the Twitter firehose for TweetReach?

This is a question we get fairly often, and although we address it in our help docs, we also wanted to address it here as it’s a little more complicated than it might seem. The short answer is yes. As for the long answer…

Twitter has two licensed data resellers - Gnip and Datasift - who can provide access to the full Twitter firehose to third parties. The full Twitter firehose includes full-coverage, real-time streaming access to all of the data from Twitter. In most cases, direct access to the full firehose is unnecessary, not to mention very expensive to consume and store. After all, as of last fall, Twitter has 215 million monthly active users, 100 million daily active users, and sees 500 million tweets per day.

So companies like us here at Union Metrics work with one of these data resellers, who have built powerful filtering tools on top of the Twitter firehose to provide high-quality access to the data we need. This makes it more efficient in both time and money for us provide the detailed, comprehensive Twitter analytics our customers want. We’ve elected to work with Gnip, and in fact are part of their Plugged In to Gnip partner program, which means they recognize that we can deliver you the highest quality Twitter data available through licensed access to the full Twitter firehose. This means you don’t have to worry about missing any data.

Our TweetReach Pro Trackers are built on Gnip’s real-time PowerTrack stream, meaning we have full access to all tweets as they are posted – with no rate limits! – for any keyword, hashtag or account you want to analyze. Similarly, TweetReach premium historical analytics are built on Gnip’s Historical PowerTrack product, and provide complete access to the Twitter archive, dating back to March of 2006. Both include full tweet coverage.

To sum up: TweetReach Twitter analytics are built from licensed access to the full Twitter firehose through Gnip.

What about Union Metrics’ other products?

Union Metrics is a certified Plugged In To Gnip partner, which means we have commercially licensed, full-coverage access to both Twitter and Tumblr data. That’s reliable, reputable data you can count on, both now and in the future. Here’s the breakdown of the data source for each of our products:

  • Our TweetReach Pro Trackers have Gnip PowerTrack access – that’s full coverage of all public tweets in real time for any search terms you enter. That means no missed tweets and no sampling.
  • Our TweetReach snapshot reports use the Twitter Search API, so they’re great for quick estimates of recent activity, but are limited to about 1500 tweets from the past week.
  • Our TweetReach premium historical analytics use Gnip’s Historical PowerTrack. That gives us full access to any public tweet in Twitter’s history, dating back to the very first tweet posted in March 2006.
  • Finally, with Union Metrics for Tumblr, we consume the full Tumblr firehose. That means we process 100% of all public posts, notes and other Tumblr activities.

If you have any other questions about our data access, please just ask!

Written by Sarah

January 29th, 2014 at 1:47 pm

Posted in Guides,Help

Tagged with , , , ,

The Week in Social Analytics #79

with one comment

It’s Friday, so that means it’s time for This Week in Social Analytics and our favorite posts of the past week in the world of measurement, analytics, and social media. See a great piece we missed? Link to it in the comments, or tell us about it on Twitter or Facebook.

Social Data Market Momentum: It’s Not About the Firehose [from Thought Experiments; written by Susan Etlinger]

“But this market, as it’s evolving, is not just a game of ‘Capture the Firehose’; it’s about taking this enormously complex, rich and challenging data set and turning it into insight that can be used to suggest trends that real people in real organizations can act on. It’s not about the firehose; it’s not even about the water. It’s about the fires the water can put out, and the things it can cause to grow.”

Nine Valuable Marketing Lessons From a Nonprofit – Charitywater.org [from KISSMetrics; written by Chloe Gray]

These lessons are equally applicable to for-profit businesses.

“Through a commitment to storytelling, beautiful design, broadcasting a unique value proposition, putting donors at the center of their strategy, and creating highly sharable content, the nonprofit has built a brand that incites the kind of loyalty, excitement, and inspiration most companies dream of.”

How To Build The Perfect Tumblr Profile For Business [from MarketingThink; written by Gerry Moran]

“Tumblr needs to be considered for every brand’s picture-perfect content marketing strategy in 2014. With native buying behavior focusing on the visual experience, brands need to be presenting their story at the multitude of customer touchpoints. This content consumption behavior demands that brands evolve into a curator and syndicator of their content assets on visual and graphical platforms like Tumblr, Instagram and Pinterest.”

4 Creative Instagram Brand Campaigns [from Business 2 Community; written by Theo Schmidt]

Check out four of the most interesting uses of Instagram for brand campaigns.

(And you can find us here on Instagram, if you’re so inclined.)

STUDY: It Only Takes 76 Seconds to Secure or Lose Brand Loyalty [from PR Newser; written by Elizabeth S. Mitchell]

“And assistance doesn’t only need to be fast — it also needs to be direct and personal. The survey revealed that 69% of consumers are more likely to be loyal to a brand that provides live, human assistance at critical moments, and 82% say their perception of a brand improves when live chat is available.”

Avoid a social crisis with one simple question [from Social Media Explorer; written by Tracey Parsons]

Before publishing something on a social account, say it out loud.

“In real time, the Out-Loud Voice litmus test it is a quick affirmation that could be the last check before something is posted online. And let’s be honest, what you post on social as a brand is very much out loud even if it is only written word. And without a doubt, most branded social media gaffes could have been prevented with this test.”

Social Business is not Dead: New charts and data reveal the real evolution of social businesses [from Brian Solis]

“Along the way, I’ve also learned that pushing for social adoption because of technology misses the point of change. The true catalyst isn’t whatever the latest trend in social media is this week. That’s reactive and almost impossible to leapfrog. The truth is that change is fueled by the affect that social media, mobile, and other forms of disruptive technologies have on customer behavior. Whether it’s B2B, B2C, B2B2C, or whatever model you prefer, as long as we’re talking about connected human beings, you can bet that social and digital in general are influencing discovery, decision-making, and impressions in every moment of truth.”

5 Tips to Help Busy Small Business Owners Better Manage their Social Media [from Eli Rose; written by Liz Jostes]

“. . .adjust your expectations to a level you can confidently maintain.”

What Do You Get With A Social Media Audit? [Top Rank Online Marketing; written by Lee Odden]

“Ultimately, the results of a social media marketing audit should reconcile the difference between the current situation and where the brand social media efforts need to be. Just like the other online marketing audits companies will perform, a social media audit is not a “one and done” situation. They should be repeated periodically as part of being accountable for the social media investment.”

B2B Experts: The Difference Between Social Media and Content Marketing [from Social Media B2B; written by Jeffrey L. Cohen]

“Your neighbor isn’t going to talk to you about those darn teenagers and their content marketing. But there’s plenty of discussion by businesses and citizens alike about social media.”

What to Do When Your Twitter Handle is Taken But Isn’t in Use [from Social Media Today; written by Deborah Sweeney]

Helpful guidelines for a 140-character crisis.

History Retweets: How Ancient Romans Created Social Media [from the Wall Street Journal; written by Tom Standage]

“Who invented social media? It wasn’t the founders of Facebook, or Twitter, or even MySpace or Friendster. Social media—the exchange of media within networks of friends and acquaintances—is much older than the Internet. A social-media environment requires two things: a certain level of literacy, and the ability to copy and deliver information cheaply and quickly. This combination first arose in the late Roman republic of the 1st century BC, more than 2,000 years ago.”

Emphasis added.

Written by Sarah

December 6th, 2013 at 9:00 am

5 questions to ask your social analytics provider

with one comment

When you choose a social analytics provider, you have the right to ask questions about the service you’re getting, even when you’re using free services. With that in mind, we wrote up a list of questions to ask when you’re checking out a new social analytics product. Got any we missed? Share ‘em in the comments below, or drop us a line.

Dog peddlers, while adorable, are not a reliable source of social analytics data. [Source: NYPL]

1. Where does your data come from?

Not all social data sources are made equally; what you can get from building a tool on a platform’s open API is vastly different from what you can get if you have access to that company’s full firehose of data. So what do you need? If you’re looking for a quick overview of recent data, then something built on an open API will work for you. If you want something more in-depth, you should consider a provider who works with a licensed data partner like Gnip or DataSift. These data resellers provide commercial, licensed access to the full data streams from platforms like Twitter, Tumblr, Foursquare and others, giving you the highest quality data possible.

Keep in mind that services built on a licensed data stream are also more reliable than something built on a free API: you don’t have to worry about hitting rate limits or missing important data. Again, if you’re just looking for enough recent information to keep track of general trends or overviews, then you don’t need to pay for extensive, real-time access to full-fidelity data– but remember the difference if your needs change.

To illustrate: if you want an idea of how many people are talking about a documentary the day after it aired and what they’re talking about, then something built on a a free API would be fine. If you made the documentary and want an extensive review of the conversation before, during and after your documentary aired and a deeper dive into the different facets of the conversation around it, you want something built on a stable, more comprehensive data source.

2. What is the firehose and do you have access to it?

A firehose is full access to all the data from a platform – that’s everything. In the case of Twitter, very few analytics providers have direct access to the full Twitter firehose, mostly because it’s unnecessary, but also because it’s quite costly. Gnip and DataSift have full firehose access, as do a very rare few others. If your analytics provider says they use the Twitter firehose, they actually probably do not. Clarify what they mean by that; the word “firehose” is misused a lot.

Instead, most serious analytics providers will have access to a full-coverage stream of data built on the firehose. This is a full-fidelity stream of tweets that matches their needs, based on a set of search queries or other filters. The result is a smaller stream of only the data they need – including all tweets that match their filters – without all the unrelated or irrelevant data.

This is a case of “you get what you pay for”; Twitter doesn’t have the infrastructure or impetus to give you access to all of their data for free, so through agreements with companies like Gnip and DataSift, a third party can gain full access to the social data they need. But this kind of data isn’t free, so be sure to choose the option that meets your needs. And if you’re using a free tool, chances are good that tool is not built on the firehose in any way.

3. What kind of data coverage do you have? Is it a sample, or the full census?

We can use Twitter as example again here, since they have several different forms of data access. Twitter’s Search API, for example, is an index of recent tweets from a window of the past few days and does not include all tweets (say, for example, you wanted an overview of what people have been searching about “overnight oatmeal in a jar” on Twitter for the past month; this wouldn’t cover your needs). You can read a more technical explanation from Twitter about the Search API here.

Other data streams are intentional portions of the full firehose, which are useful for sampling and other use cases. Twitter has a decahose option, for example, that includes a random sample of 10% of all tweets. It’s great for research, but not ideal if your needs require full-fidelity coverage.

The only full-coverage options are through a data provider like Gnip, or from a partnership with the platform itself. This could be through the full firehose (which only a couple companies actually have), or through a full-coverage, keyword-based data stream. Ask your analytics provider if you’ll have full-coverage access to your tweets, or if they use just a sample.

4. Does the data comply with the platform’s terms of service (ToS)?

The great unread novel of our time is the complete terms of service to just about anything. You’ll want to do your homework with your data provider, however, and be sure that their product does indeed comply with the ToS of your platform of choice. An easy way to do this is to check and see if they are a partner with them, or an approved or preferred provider. You can also check with the data resellers like Gnip for this. You’ll also want to be sure it says this on the platform’s website, and isn’t just a wild, false claim on the data provider’s. If both sites say they work together, it’s a safe bet they’re following the ToS, or the platform wouldn’t have partnered with them or given them a title of approval. If it seems unclear, don’t be afraid to ask for clarification. If they’re not willing to talk about it, go elsewhere so you won’t run the risk of your provider being shut down and disappearing with all of your analytics.

5. Are the metrics actual counts or just estimates?

Finally, even if your provider has access to high quality data, you want to be sure they’ve built a tool that gives you the best possible measure of the specific data you’re looking for. (You want the best results around “overnight oatmeal in a jar”, not “overnight oatmeal in a crockpot”, after all.) If you test several different providers and get wildly different results, compare those results with how these companies are telling you they generate their results. If they don’t have documentation that tells you how their tool works, or that documentation is vague and confusing, that’s a bad sign.

If in comparing results from two companies that are both built on the Twitter Search API, you notice one is returning wild estimates and the other is giving you the most accurate count they can, definitely go for the latter. Don’t go for the tool that returns estimates just because the numbers are bigger. You don’t want your marketing plan or quarterly report to be based on imaginary numbers.

Bonus: 6. What data access does Union Metrics have?

We are a certified Plugged In To Gnip partner, which means we have commercially licensed, full-coverage access to Twitter and Tumblr data. That’s reliable, reputable data you can count on, both now and in the future. Here’s the breakdown.

  • Our TweetReach Pro Trackers have Gnip PowerTrack access – that’s full coverage of all public tweets in real time for any search terms you enter. That means no missed tweets and no sampling.
  • Our TweetReach snapshot reports use the Twitter Search API, so they’re great for quick estimates of recent activity, but are limited to about 1500 tweets from the past week.
  • Our TweetReach premium historical analytics use Gnip’s Historical PowerTrack. That gives us full access to any public tweet in Twitter’s history, dating back to the very first tweet posted in March 2006.
  • Finally, with Union Metrics for Tumblr, we consume the full Tumblr firehose. That means we process 100% of all public posts, notes and other Tumblr activities.

Have any questions about our data access? Please just ask!

Written by Sarah

October 9th, 2013 at 11:14 am