Andreas Weigend
Data Mining and E-Business: The Social Data Revolution
Spring 2009
STATS 252
Stanford University

This page has some of the emails sent out to students

9. Next week: Mon 2:30-5 last office hours, Tue noon funny video submission deadline, Wed noon class on privacy and data mining

Sent: June 3, 2009 by Andreas Weigend++
Note: I forgot to attach tothe email the pdf of Ustream metrics, now uploaded

Dear Stats 252 students,

Three quick things for next week:
  1. Monday June 8, 2:30 – 5:00pm : Office hours
  2. Tuesday June 9, noon : Deadline for the short, funny videos for the five $200 prizes
  3. Wednesday June 10, 12:15 – 3:05pm (Gates B01, same room). Class, focus on privacy.

Ad 1) Since several people asked: I will be holding office hours this coming Monday, June 8, from 2:30- 5 (basically the usual class time) in my office in the Stats Department: Please come by with any questions, feedback, suggestions and ideas. Please self-organize by putting down your name behind a time on http://stanford2009.wikispaces.com/Wishes+by+students so people don’t have to wait. And if you have anything you want to see addressed in the last class, it is the same page. Your input is very important to me.

Ad 2) I will give out up to five cash prize of $200 each ($1000 total) for short, funny videos you create that I will use in at least one of my talks this year (http://weigend.com/speaking).
Deadline: Tuesday June 9, noon. (This gives us 24 hours before the last class to select the ones to show in class). You need to:

Ad 3) Our last class will focus on privacy in the context of data mining and marketing. We are fortunate to have two great speakers who will share their insights with us:
  • Cynthia Dwork (Microsoft Research) will bring some shocking examples of how easy it is to crack anonymized data when there are auxiliary data. Check out the video of her brilliant talk last year http://www.almaden.ibm.com/institute/2008/resources/2008/videos/17.wmv .
  • Amit Goswami (Orange-FT Research) will first reflect why our initial ideas on mobile marketing were all wrong – people were talking more than a decade ago how the mobile phone will revolutionize marketing, and then outline how he sees its future.
Amit's company has also generously offered $500 for lunch, and Robert Clegg (rclegg@millionstudentmarch.com) has kindly agreed to get stuff.

Good luck on your finals, see some of you during office hours on Monday, and everybody on Wednesday!

-- Andreas

PS: I watched part of the Ustream experiment from this week. The sound on Ustream was quite distorted, sometimes hard to make out what people were saying. It is important to understand what and when video really adds. It clearly did around minute 20 or so in the first part http://www.ustream.tv/recorded/1592981 where we discuss the graph John brought. The second part is at http://www.ustream.tv/recorded/1593287. I am attaching the metrics of number of people, minutes watched etc. While we announced it on FB and twitter, we only had a couple of people watch it for long enough to get anything out of it. My hope of getting metadata through public annotations in real time, instead of the private notes people take for themselves, is still in the future. On the other hand, you really exceed my expectations with the course wiki :)


8. Monday Jun 1 right after class food and drinks in Sequoia Hall, five $200 prizes for funny SDR videos and more...

Sent: May 30, 2009 by Andreas Weigend

Dear Stats 252 students,

Exciting news: We’ll have John Carnahan, VP Chief Scientist, Audience Network at Fox Interactive Media (FIM) in class on Monday. He is incredibly smart and probably the best guy in the world for online ads, hypertargeting, and monetization.
His company, FIM MySpace, is sponsoring a reception right after class. Besides drinking, eating yummy Jamaican food from Mango Café, mingling and getting to know each other a bit, it will be an opportunity to thank you for your hard work this quarter.

To make this a success, we need your help:
  • RSVP on facebook : Please let us know by Sunday 5pm that you are coming via RSVP at
    http:/www.facebook.com/event.php?eid=87957816823 . And invite your friends who are interested.
  • Volunteers for reception: We do need volunteers to get drinks and set things up, ideally also a bartender for those above 21. Please email enrique.allen@stanford.edu. BTW, today is Enrique’s birthday!

Also, to move the discussion forward:

In class on Monday, II will be announcing the [[Emails+to+students#video|SDR Video Contest]]. Here is the preview: I will give out up to five cash prize of $200 each ($1000 total) for short, funny videos you create that I will use in at least one of my talks this year (http://weigend.com/speaking). Besides English, it would be great to have Italian, Spanish (Mexico, Argentina), Portuguese (Brazil), and of course Putonghua and Singlish :) since these are the countries I will be in.
Deadline: Tuesday June 9, noon. (This gives us 24 hours before the last class to select the ones to show in class)
You need to:

Why? We want your creativity: interview a friend, roommate, your prof, your parrot, your prof, in any funny situation. Play with twitter, share a story who you met through MrTweet, or what your friends said when you showed them your recommendations. Anything funny coming out of a dating site, or about something you bought because of social commerce that you totally didn't need / that you can't live without any more. Show the world how social discovery is so totally different from traditional marketing or lame Facebook ads. Give an example of what twitter does for you, or how it annoys the hell out of you. Show how you use Facebook, Craigslist etc to get paid/laid. Whatever. Needs to work for international marketers and executive who have never heard of twitter, and should realize just how much they are totally out of it, in a funny way. While entirely voluntary we hope it gives you the chance to think through data in the world we live in from yet another angle, and to express it in a new way. It does not need to be original work, anything from http://www.youtube.com/watch?v=pRlrx7Dg7Dc to http://www.youtube.com/watch?v=bkSaNToDbW8 or http://www.youtube.com/watch?v=PN2HAroA12w.

Anything else? The teaching team passionate about this course and really want you to have a great experience. Let us know what you want! We created http://stanford2009.wikispaces.com/page/history/Wishes+by+students two weeks ago and it still is empty! This is your space, use it.

Looking forward to class on Monday and to getting to know you a bit better at the party! Please RSVP by Sunday 5pm. and let your friends know.

--Andreas

PS: Note the optional homework assignments for extra credit (especially those who struggled in the Delicious and Twitter assignments) and sign up ASAP because there are only a total of 15 slots on http://stanford2009.wikispaces.com/Survey_SetA, etc. Enrique is about to follow up with an email with details.


7. Email about extending HW6 (Twitter) by one week and making HW7 (Yobo) optional

Sent: May 27, 2009 by Andreas Weigend

Dear STATS 252 students,

Quick update: First, I want to thank Enrique Allen, Ryan Mason and Ron Chung for their amazing help with class, and especially for picking up the work Feng Zheng did not manage. FYI, Feng has been relieved of his duties by the department chair last weekend.
I apologize for the problems with support – you saw my surprise when I discovered during last class that the email I sent did not reach several of you. I also apologize for the delayed feedback on your assignments. Please continue to email stats252.spring2009@gmail.com to reach Enrique, Ron and Ryan. They are working super hard and do the very best anyone can in a seriously understaffed situation.

I only heard good things about the twitter hack-a-thon last night, almost 20 students came and got help from peers, including Chris Anderson, Emile Chamoun, Carlin Eng, Jeff Mellon, and Mike Polcari. Thanks! I could not be there since I am at D7 http://www.allthingsd.com in San Diego, where I talked last night to Twitter’s Ev Williams about what you are doing in class.

Given that it took longer than expected to get everybody whitelisted for the twitter API, here are two updates:
1) We extended the deadline for HW6 to Thursday June 4, 2009, 5pm. This also gives you time to incorporate what you learned last night, while still giving us time to grade and give you feedback at our last class session (Wednesday, June 10, 12:15-15:05, on Privacy).
2) We made HW7 (analyzing the Yobo music DNA data) entirely optional. It is up the wiki if you are curious, but please don’t worry about it,. I will explain my thinking that lead to this homework in class on Monday (June 1, 14:15 – 17:05, on Ads).

I am looking forward to finishing strong together, and creating learnings that endure beyond class. I have created http://stanford2009.wikispaces.com/Wishes+by+students and ask you to please just down any wishes you have for the rest of the quarter.

You really are doing cool stuff -- I believe no other course in the world has similar innovative problem sets. Looking forward to seeing you on Monday,

Andreas


6. Email about prediction markets

Dear Stats 252 students,

We thought it would be interesting for students to get hands-on experience with prediction markets.
This exercise will be for extra credit because we realize there are other assignments due in the next few weeks.

Here is the setup:

1) Prediction market exercise starts this coming Monday May 25 and ends on Monday June 8.

2) Final results will be announced in class on June 10.

3) This is for extra credit.

4) You should have received an email invite for SDR.INKLINGMARKETS.COM

5) We set up two "practice" predict market games for those that want some practice before official "start".
These two simple "games" end on Monday May 25.

6) Official games will be set up that are related to the Social Data Revolution. This may include predicting how your SDR assignments will perform.

=

=Try to make the most money among all your peers. It'll be a fun and worthwhile experience.
=

=

=INSTRUCTIONSYou should receive an invitation from Inkling to join the SDR Inkling Prediction Market.Please sign in and you should be able to start playing.


You can find the two games here: https://sdr.inklingmarkets.com/pending/markets

=

=

=Here is some quick information about the "practice" games:


=

Market 1:Starting Price: $50 A price of $50.00 means there is currently a 50.0% chance this will occur.
description:
Will the Ralph Lauren stock price increase (relative to its closing price on May 26, 2009) following the release of the earnings report on May 27, 2009? In other words, will the stock price at 4:00pm EST on May 27, 2009 be strictly higher than the closing price at 4:00pm EST on May 26, 2009.

Ralph Lauren's earnings report will be release at 8:00 AM EST. This prediction market will end on May 25, 2009 at midnight.

links to Ralph Lauren's stock price: google finance: http://www.google.com/finance?client=ob&q=NYSE:RL

yahoo finance: http://finance.yahoo.com/q?s=rl

=

Market 2: Who will win game 4 between the Lakers and Nuggets in the NBA playoffs?

description: The LA Lakers are facing the Denver Nuggets in Western conference finals of the NBA. Game 4 will take place on Monday, May 25, 2009 at 6pm. Who do you think will win game 4? Our prediction market will close on Sunday, May 24, 2009 at midnight PST.

Starting Price:Lakers $50, Nuggets $50 $50
A price of $50.00 means there is currently a 50.0% chance this will occur.


5. Email about HW4 (Delicious) and HW6 (Twitter)


Dear STATS 252,

delicious HW suggestions:
1. This should be done individually, but feel free to discuss questions about Python programming
2. Reference Collective Intelligence by Toby Segaran http://proquest.safaribooksonline.com/9780596529321
I could not access this book in its entirety (as an off campus, SCPD student) until I signed up for the free 10-day trial. E-mail me if you need a copy of Ch2. - Sylvie B. (sylviebryant@gmail.com)

The Twitter HW is likely to take longer than the delicious HW. We are going to hold special office hours with food because many of you have voiced concern. However, since our teaching team is understaffed we are offering extra credit to experienced python developers who are willing to help their classmates. We are planning to host the hack-a-thon Tuesday, May 26th from 7-9pm at the d.school Bldg 524. If you have friends willing to help outside our class, we can offer $20/hour + food.

If you are a developer interested in helping please respond to enrique.allen@stanford.edu.

Thank you,
Enrique + Ryan+ feng!


4. Email about Class 7

Sent: Mon 5/18/2009

Dear Stats 252 Students,
Thanks for the positive feedback on my email last Friday Sorry about incorrect youtube link, now correct on http://stanford2009.wikispaces.com/Emails+to+students.

Class today is action packed:

1. Lecture on recommender systems.

2. Brief discussion of first insights of the survey
Just started to look at http://stanford2009.wikispaces.com/HW5_SurveyInsights Will pull up during class what we have then, and will give a few students the chance to comment on what they put on it. Quality of work counts, boiling things down to the essence is key. BTW, this also applies when we ask you on the Twitter assignment to reflect – reflecting on what has changed in your mind is an important of how people learn.BTW, while I love good visualizations, simple pie charts often don’t add much, giving the percentages is often much cleaner and help us see the bigger picture as we look at the entire page.

3. Introducing Twitter people recommendation homework http://stanford2009.wikispaces.com/HW6_Twitter
Will be joined by Nick Kallen from Twitter, Yu-Shan Fung from Discoverio, and Nova Spivack
http://novaspivack.typepad.com/nova_spivacks_weblog/2009/02/twitter-changes-everything-the-world-just-got-faster.html See also Erik Schonfeld’s post on “real time search” http://www.techcrunch.com/2009/02/15/mining-the-thought-stream/

4. The promise vs reality of the “intelligent web” (aka sematic web).
Talk by Nova Spivack, the founder and CEO of Radar Networks, a semantic Web startup that operates the personal information-organizing service Twine. He is a noted authority on the semantic Web, artificial intelligence, and is a space enthusiast.
In 1994, he co-founded EarthWeb, where he was a board member and served as Executive Vice-President for Products, Strategy and Marketing. Prior to that, he worked with computing pioneers Danny Hillis at Thinking Machines and Ray Kurzweil at Kurzweil Computer Products.
BTW, his grandfather was management guru Peter Drucker.


I also updated our wiki home page http://stanford2009.wikispaces.com/ . If those of you who had emailed me about what you are willing to help with could be so kind and enter your info on http://stanford2009.wikispaces.com/HELP+with+class I would be sure that I didn’t miss anyone.

See you soon,
Andreas



3. Status email to people in class (registered, auditing and friends)

Sent: Fri 5/15/2009

Dear Stats 252 Students,
A quick email about where we are, two thirds through the quarter.

We’ve looked at a breadth of material from the evolution of passive data (“sniffing the digital exhaust”) to actively instrumenting people and the environment. To help you understand (and experience) the challenges of social data, we assigned the Facebook Pages HW. I know that this was not easy, but most of you did a great job. BTW, for the ad parts, we will revisit machine learning for ads in class on June 1.

We then turned to the basics of analytics, and surveyed some of the now easily available data sources. But the main focus has to remain on how data and models influence decisions. I found the MS&E presentation on Decision Analytics very useful, and hope that you also appreciated the evolution towards engaging the crowd via predictive markets.
Itamar’s talk gave us an inside scoop into how Facebook processes petabytes of data and designs experiments that model user behavior. It was fun to come up with hypotheses, and see from Eric how the traditional model of “influencer marketing” seems not to apply to FB with its ultra-light weight interactions any more.
This week, Reid and DJ hopefully shared their insights into analytics and visualization, and showed when edge cases and being clever is important. I was impressed by how analytics drives product development at LinkedIn and enables quick evolution of products -- fast iteration rules.
This Monday, May 18, I will discuss recommender systems. June 1 focuses on machine learning for advertising with George John, CEO of RocketFuel. In the last class, Wednesday June 10, 12:15 – 15:05 (instead of the final), we will discuss privacy with Cynthia Dwork.
The current assignments should not be too much work.
The final two problem sets will hopefully be fun as well (will be up the day they are assigned):
  • HW6 (out May 18, due May 28) asks you to design a recommender system for Twitter suggesting people you might be interested in, http://stanford2009.wikispaces.com/HW6_Twitter Please contact the teaching team if you're interested in helping. We will have the people behind MrTweet come to class this Monday.
  • HW7 (out May 25, due June 4) concludes the quarter by giving you data on the music DNA data of http://www.yobo.com/, a company in Beijing, where copyright seems less of an issue than here J
And, last but not least, thank you for the AMAZING work you have been doing on our course wiki!! I really am impressed by what you manage to create there.
Have a great weekend, and see you Monday at 2:15!
Thanks!
Andreas

PS: Since we are way understaffed with TAs, if anyone wants to help out including possibly insights for the blog, please do let me know. Made a page on the wiki, stanford2009.wikispaces.com/HELP+with+class
You might also want to check out the short interviews done at the public school at the other side of the Bay…
youtube.com/user/socialdatarevolution



2. Email about 2009 course to former students

Sent: Mon 3/9/2009
STATS 252 - Spring Quarter 2009
Data Mining and Electronic Business -- The Social Data Revolution


Dear former STATS 252 students,
I hope you are well and working on cool projects. I'm putting together what promises to again be an exciting Stats252 course. Actually, o make this year's course even better I need your help to spread the word about the course. Therefore, I would like to ask you for 5 minutes of your attention:
  1. Please join our Facebook page "Social Data Revolution", and post on the wall something you got out of the course when you took it.
  2. Invite the smartest students you know to join that Facebook page and give them a reason to to sign up for STATS 252 this Spring (Axess ID 10584).
  3. Remember everyone is welcome, so feel free to come to any class that interests you. We will have interesting speakers again. If you want to add the course schedule to your own calendar, please note that the class is one hour earlier that last year, i.e. Mondays 2:15 - 5:05, Gates B01, starting April 7th.
  4. I would also like to invite you and/or your friends to the information session / preview lecture on Monday, March 9, 2009, 2:15 – 3:05 at Skilling Auditorium.
Since last year, the creation and distribution of social data has vastly increased, but we are still facing the problems of information overload and of discovery of relevant content and interesting people. Social marketing is more important than ever, this is why I am asking you to talk to your friends about the class.Thanks in advance!
And as always, please let me know if anything exciting is happening, and also if there is anything I can do for you. And if you can’t make it to class, I continue to put up mp3s after each class, and have high expectation for this course wiki. New is that I will also have transcripts of each class on the web, and a weekly short videos related to the course (some as interviews with guest speakers). Subscribe to http://youtube.com/user/socialdatarevolution .

Cheers,
Andreas Weigend | +1 650 906-5906 | weigend.com

PS: Let me know if you have any recommendations for startups in this area, and/or have heard any great speakers. We have for this year Reid Hoffman (CEO of LinkedIn), Jia Shen (CTO of RockYou), George John (CEO of RocketFuel, using machine learning for ad serving), Nova Spivack (CEO of Radar Networks / Twine), Mingyeow Ng (CEO of Discoverio / MrTweet) and Cynthia Dwork (brilliant researcher at Microsoft on data mining and privacy). On the more practical side, we have Nick Kallen (Twitter) and Itamar Rosenn (Facebook) being available for questions and ideas you might have given the two homeworks we are doing on those platforms, and three Stat252 alumni will share what they have learned and done since, Linus Liang (giving reasons for his successes first with Facebook Apps, then with iPhone apps), Bo Cowgill (talking about Google's internal predction markets), and Eric Sun (showing how information spreads on Facebook and discussing its implications to marketing).


1. Email about 2009 course to departments

Sent: March 2009 to friends in CS, MS&E, and GSB asking to forward to admin to forward to prospective students
STATS 252 - Spring Quarter 2009
Data Mining and Electronic Business -- The Social Data Revolution


Class time: Mondays 2:15 - 5:05 (starting April 6, 2009), Gates B01.
Registration: STATS 252, Axess ID 10584.

Extract Insights from Twitter
Put Yourself in Google's Shoes
Develop Relevance Beyond Amazon
Build Revolutionary Facebook Applications
In the last year, your location data and personal medical information have become the latest streams in the torrent, joining email, clicks, searches, social networking, and buying patterns. This course will dramatically change how you think about your river of data.

How can these new data sources make our lives easier, more effective, more interesting? How can we get better recommendations, based on our behavior and the behavior of our friends? How can reputation systems help with decisions about who to trust?

Gathering, sharing, and storing data has become trivial. But what shall we collect, and what applications can we build that users really want?

Moving beyond graph and guess, push and pray, launch and learn, and so on, this course gives you tools and strategies for successful applications. How can you optimize their virality, and spot weaknesses early? How can you entice users to interact with the app, and recommend it to their friends?

Each class is structured according to PHAME: define relevant Problems, invent Hypotheses, create Actions, design Metrics, and conduct Experiments. We also introduce a key driver to encourage users to provide critical data: Return on Personal Engagement (ROPE). Users who gain a benefit (tangible or psychological) from participating are far more likely to do so, and we discuss how to design incentives to encourage participation.

In addition to discussing applications that succeeded, we also discuss applications that failed, and try to distill out the reasons for success and failure. This course also includes highlights from two courses I developed and taught at Berkeley in 2008, "Marketing 2.x" and "The Social Data Revolution".

STATS 252 it taught by Andreas Weigend (former Chief Scientist of Amazon.com) who shares his first-hand experiences of working with some 100 companies over the last 15 years.Previous guest speakers include Jeff Hammerbacher (former head of data at Facebook), Reid Hoffman (founder of LinkedIn), Jan Pedersen (former chief scientist of Yahoo), Joshua Schachter (founder of Delicious), Paul Muret (Google Analytics), Johann Schleier-Smith and Greg Tseng (founders of Tagged).

The course is open to undergraduates and graduate students with experience in web programming. They are expected to actively engage in class discussions, to have their assumptions challenged, and to bring their diverse backgrounds to bear. After each class, a detailed write-up is created by the students as the course wiki.

Hands-on assignments include understanding and leveraging data and analytics on the web analytics, defining robust variables and metrics and that characterize engagement on a Facebook pages, designing several contracts for a prediction market related to class, subsequently participating in its inkling implementation, creating a recommender system for web content based on delicious data (code supplied in python)

Further possibilities are applying geolocation to dating, creating a recommender system for Twitter
Questions? See www.weigend.com/teaching/stanford More questions? Email Andreas Weigend