5 July 2012 18 Comments

How to Harvest Facebook updates with Python, MySQL or SQL Server

While most people focus on search.twitter.com (or automated pulls from their API) for data collection regarding the "pulse" on a particular topic or brand - they often overlook the fact that the public Facebook timeline can be a useful source of data as well.

That's how I was able to harvest both twitter and facebook streams during the Stanley Cup games and look for similarities (like, tons).

The BOOK OF FACE!

Here I am going to show you 2 quick and dirty scripts will gather data from the Facebook API and insert it into a database (using MySQL or Microsoft SQL Server).

First, the SQL tables..

(don't fret if your cut and paste skills are shitty, I've got zip files at the bottom of the post)

MySQL

SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";

--
-- Database: `facebank`
--

-- --------------------------------------------------------

--
-- Table structure for table `fbsearchlog`
--

CREATE TABLE IF NOT EXISTS `fbsearchlog` (
  `RunId` INT(11) NOT NULL AUTO_INCREMENT,
  `BatchId` INT(11) DEFAULT NULL,
  `RunDate` datetime DEFAULT NULL,
  `Keyword` VARCHAR(50) DEFAULT NULL,
  `HarvestedThisRun` INT(11) DEFAULT NULL,
  `TotalHarvested` INT(11) DEFAULT NULL,
  `RunTime` FLOAT DEFAULT NULL,
  PRIMARY KEY (`RunId`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=338 ;

-- --------------------------------------------------------

--
-- Table structure for table `searchbank`
--

CREATE TABLE IF NOT EXISTS `searchbank` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `run_id` INT(11) DEFAULT NULL,
  `fid` VARCHAR(250) DEFAULT NULL,
  `from_name` VARCHAR(250) DEFAULT NULL,
  `from_id` BIGINT(20) DEFAULT NULL,
  `keyword` VARCHAR(50) DEFAULT NULL,
  `type` VARCHAR(50) DEFAULT NULL,
  `posted` datetime DEFAULT NULL,
  `message` VARCHAR(5000) DEFAULT NULL,
  `story` VARCHAR(5000) DEFAULT NULL,
  `link` VARCHAR(1500) DEFAULT NULL,
  `likes` INT(11) DEFAULT NULL,
  `comments` INT(11) DEFAULT NULL,
  `shares` INT(11) DEFAULT NULL,
  `harvested` datetime DEFAULT NULL,
  `page_likes` INT(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1 AUTO_INCREMENT=12255 ;

-- --------------------------------------------------------

--
-- Table structure for table `searches`
--

CREATE TABLE IF NOT EXISTS `searches` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `run_id` INT(11) DEFAULT NULL,
  `fid` VARCHAR(250) DEFAULT NULL,
  `from_name` VARCHAR(250) DEFAULT NULL,
  `from_id` BIGINT(20) DEFAULT NULL,
  `keyword` VARCHAR(50) DEFAULT NULL,
  `type` VARCHAR(50) DEFAULT NULL,
  `posted` datetime DEFAULT NULL,
  `message` VARCHAR(5000) DEFAULT NULL,
  `story` VARCHAR(5000) DEFAULT NULL,
  `link` VARCHAR(1500) DEFAULT NULL,
  `likes` INT(11) DEFAULT NULL,
  `comments` INT(11) DEFAULT NULL,
  `shares` INT(11) DEFAULT NULL,
  `harvested` datetime DEFAULT NULL,
  `page_likes` INT(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MEMORY DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

Note that one table "searches" is a MEMORY table - that's because its just used as a temp table to avoid dupes. We don't need to hold on to that data, and in-memory tables are hella fast.

SQL Server

In this case 2008R2, but any semi-recent version will work (besides SQL Server 2008 Express is a great database, and is freaking free).

USE [fbOpenGraph]
GO

/****** Object:  Table [dbo].[fbsearchLog]    Script Date: 07/05/2012 14:22:27 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[fbsearchLog](
    [RunId] [INT] IDENTITY(1,1) NOT NULL,
    [BatchId] [INT] NULL,
    [RunDate] [datetime] NULL,
    [Keyword] [VARCHAR](50) NULL,
    [HarvestedThisRun] [INT] NULL,
    [TotalHarvested] [INT] NULL,
    [RunTime] [FLOAT] NULL,
 CONSTRAINT [PK_fbsearchLog] PRIMARY KEY CLUSTERED
(
    [RunId] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

USE [fbOpenGraph]
GO

/****** Object:  Table [dbo].[SearchBank]    Script Date: 07/05/2012 14:22:39 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[SearchBank](
    [id] [INT] IDENTITY(1,1) NOT NULL,
    [run_id] [INT] NULL,
    [fid] [VARCHAR](250) NULL,
    [from_name] [VARCHAR](250) NULL,
    [from_id] [BIGINT] NULL,
    [keyword] [VARCHAR](50) NULL,
    [TYPE] [VARCHAR](50) NULL,
    [posted] [datetime] NULL,
    [message] [VARCHAR](MAX) NULL,
    [story] [VARCHAR](MAX) NULL,
    [link] [VARCHAR](1500) NULL,
    [likes] [INT] NULL,
    [comments] [INT] NULL,
    [shares] [INT] NULL,
    [harvested] [datetime] NULL,
    [page_likes] [INT] NULL,
 CONSTRAINT [PK_SearchBank] PRIMARY KEY CLUSTERED
(
    [id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

USE [fbOpenGraph]
GO

/****** Object:  Table [dbo].[Searches]    Script Date: 07/05/2012 14:22:48 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[Searches](
    [id] [INT] IDENTITY(1,1) NOT NULL,
    [run_id] [INT] NULL,
    [fid] [VARCHAR](250) NULL,
    [from_name] [VARCHAR](250) NULL,
    [from_id] [BIGINT] NULL,
    [keyword] [VARCHAR](50) NULL,
    [TYPE] [VARCHAR](50) NULL,
    [posted] [datetime] NULL,
    [message] [VARCHAR](MAX) NULL,
    [story] [VARCHAR](MAX) NULL,
    [link] [VARCHAR](1500) NULL,
    [likes] [INT] NULL,
    [comments] [INT] NULL,
    [shares] [INT] NULL,
    [harvested] [datetime] NULL,
    [page_likes] [INT] NULL,
 CONSTRAINT [PK_Searches] PRIMARY KEY CLUSTERED
(
    [id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

Okay - now that our data structure is in place - let's get some Pythonin' goin' onin'!

First the MYSQL version

(will work on Windows and Linux)
Note: you're going to need to following Python (2.7 32-bit recommended) modules to use this bad boy as-is:

import string, json, pprint
import urllib
from datetime import timedelta
from datetime import date
from time import *
import string, os, sys, subprocess, time
import simplejson as json

from time import time, sleep
startTime = time()

from colorama import init
init() #init colorama (needed on win32, not in *nix)
from colorama import Fore, Back, Style

import MySQLdb

pp = pprint.PrettyPrinter(indent=3)

conn = MySQLdb.connect(host='localhost',  user='root', db='facebank', passwd='harvie') #, passwd='fsdfsdfsdf'
conn.autocommit(True)
cur = conn.cursor()

cur.execute("select ifnull((select ifnull(run_id,0) from searchbank order by run_id desc limit 0,1),0)")
run_id = cur.fetchall()[0][0]+1
print 'BatchRun #' + str(run_id)

newones = 100
keywords = ['metallica','james hetfield', 'lars ulrich', 'kirk hammett', 'rob trujillo', 'jason newsted','cliff burton','tsoukalos','ancient aliens','bigfoot','vaynermedia']
# put yer keywords here!

for keyword in keywords:

    sleep(1)
    nextpage = 'https://graph.facebook.com/search?q="'+str(keyword)+'"&type=post&limit=100'
    newones = 100

    while newones > 0 and nextpage <> 'NO':
        sleep(1)
        f = urllib.urlopen(nextpage)
        s = f.read()
        f.close()
        ss = json.loads(s)
        #pp.pprint(ss)

        try:
            for num in range(0,len(ss['data'])):
                #print '*************************'
                #for majorkey, subdict in ss['data'][num].iteritems(): #'paging' or 'data'?
                #   print majorkey
                #   print subdict
                #   for subkey, value in subdict.iteritems():
                #       print subkey, value
                #print '  '+str(ss['data'][num]['created_time'].encode('ascii', 'ignore'))
                createdtime = str(ss['data'][num]['created_time'].encode('ascii', 'ignore'))
                #print '  '+str(ss['data'][num]['type'].encode('ascii', 'ignore'))+'!'
                fbtype = str(ss['data'][num]['type'].encode('ascii', 'ignore'))
                try:
                    #print ' msg '+str(ss['data'][num]['message'].replace("'","''").replace(';','').encode('ascii', 'ignore'))+'!'
                    fbmessage = str(ss['data'][num]['message'].replace("'","''").replace(';','').encode('ascii', 'ignore'))
                except:
                    fbmessage = ''
                try:
                    #print ' sty '+str(ss['data'][num]['story'].replace("'","''").replace(';','').encode('ascii', 'ignore'))+'!'
                    fbstory = str(ss['data'][num]['story'].replace("'","''").replace(';','').encode('ascii', 'ignore'))
                except:
                    fbstory = ''

                try:
                    #print '  '+str(ss['data'][num]['comments']['count'])+'.'
                    fbcomments = str(ss['data'][num]['comments']['count'])
                except:
                    fbcomments = 0
                try:
                    #print '  '+str(ss['data'][num]['likes']['count'])+'.'
                    fblikes = str(ss['data'][num]['likes']['count'])
                except:
                    fblikes = 0
                try:
                    #print '  '+str(ss['data'][num]['shares']['count'])+'.'
                    fbshares = str(ss['data'][num]['shares']['count'])
                except:
                    fbshares = 0
                try:
                    namee = str(ss['data'][num]['from']['name'])
                except:
                    namee = ''
                try:
                    #print '  '+str(ss['data'][num]['link'])+'.'
                    link = str(ss['data'][num]['link']).replace("'","''")
                except:
                    link = ''
                #print '*************************'

                #print str(ss['data'][num]['id'])
                fid = str(ss['data'][num]['id'])
                print str(ss['data'][num]['id']) + ' - ' + namee + ' (' + fbtype + ')'
                from_name = namee.replace("'","''").encode('ascii', 'ignore')
                #print str(ss['data'][num]['from']['id'])
                from_id = str(ss['data'][num]['from']['id'])

                query = """insert into searches (fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id)
                    values ('"""+str(fid)+"""', '"""+str(from_id)+"""', '"""+str(from_name)+"""', '"""+str(keyword)+"""', '"""+str(fbtype)+"""', '"""+str(link)+"""',cast('"""+str(createdtime).replace('+0000','')+"""' as datetime), '"""+str(fbmessage)+"""', '"""+str(fbstory)+"""', '"""+str(fblikes)+"""', '"""+str(fbcomments)+"""', '"""+str(fbshares)+"""', now(), '"""+str(run_id)+"""')"""
                #print query
                try:
                    cur.execute(query)
                except:
                    print '#################### SQL ISSUE ###########################'
                    print query
                    print '#################### SQL ISSUE ###########################'
                #print '  '+str(ss['data'][num]['message'].encode('ascii', 'ignore'))+' '
                conn.commit()
            #print ss['paging']['next']
            print str(len(ss['data'])) + ' = element "data" length'
            #sleep(1)

            cur.execute("""insert into searchbank (fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id)
            select distinct fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id from searches where fid NOT in
            (select distinct fid from searchbank)""")

            cur.execute("""delete from searches where keyword = '"""+str(keyword)+"""'""")

            cur.execute("select ifnull((select ifnull(TotalHarvested,0) from fbsearchlog where keyword = '"+str(keyword)+"' order by rundate desc limit 0,1),0)")
            total_harvested = cur.fetchone()[0]

            cur.execute("""insert into fbsearchlog (BatchId, keyword, RunDate, HarvestedThisRun, TotalHarvested) values
            (
            '"""+str(run_id)+"""',
            '"""+str(keyword)+"""',
            now(),
            ((select count(*) from searchbank where keyword = '"""+str(keyword)+"""')-"""+str(total_harvested)+"""),
            (select count(*) from searchbank where keyword = '"""+str(keyword)+"""')
            )""")

            conn.commit()

            cur.execute("""select ifnull(HarvestedThisRun,TotalHarvested) from fbsearchlog where BatchId = '"""+str(run_id)+"""' and keyword = '"""+str(keyword)+"""' order by RUnDate desc limit 0,1""")
            harvienum = cur.fetchone()
            newones = harvienum[0]
            print Fore.GREEN + Style.BRIGHT + str(keyword) + Style.RESET_ALL
            print str(newones) + ' actually kept'

            try:
                nextpage = ss['paging']['next']
                print nextpage
            except:
                nextpage = 'NO'
                print 'No next page avail'
        except:
            print 'Some kind of API errors - invalid data JSON returned maybe?'
            nextpage = 'NO'
            newones = 0
            print 'ERROR'
            print sys.exc_info()[0]
            print sys.exc_info()[1]
            sleep(2)
    #sleep(2)



elapsed = time() - startTime
print "Finished. Total time: " + strftime('%H:%M:%S', gmtime(elapsed))
done_time = strftime('%H:%M:%S', gmtime(elapsed))

Note: the only stuff you SHOULD have to change is the MySQL connection info and your list of keywords (unless you really want to harvest all the Facebook mentions of 'lars ulrich', in which case - knock yourself out!)

Now the SQL Server version

Note: you're going to need to following Python (2.7 32-bit recommended) modules to use this one as-is:

import string, json, pprint
import urllib
from datetime import timedelta
from datetime import date
from time import *
import string, os, sys, subprocess, time
import simplejson as json
#import winsound

from time import clock
import time
clock()


import pymssql

pp = pprint.PrettyPrinter(indent=3)

conn = pymssql.connect(host='localhost:1450',  user='sa', password='harvie',database='fbopengraph')
conn.autocommit(True)
cur = conn.cursor()

cur.execute("select isnull((select top 1 isnull(run_id,0) from fbopengraph.dbo.searchbank order by run_id desc),0)")
run_id = cur.fetchall()[0][0]+1
print run_id


newones = 100
keywords = ['bic','metallica','james hetfield', 'lars ulrich', 'kirk hammett', 'rob trujillo', 'jason newsted','cliff burton','garyvee','vaynerchuk','tsoukalos','ancient aliens','bigfoot','vaynermedia','orion']

for keyword in keywords:

    time.sleep(1)
    nextpage = 'https://graph.facebook.com/search?q="'+str(keyword)+'"&type=post&limit=100'
    #homelink = 'https://graph.facebook.com/'+str(fid)
    newones = 100



    while newones > 0 and nextpage <> 'NO':
        time.sleep(2)
        f = urllib.urlopen(nextpage)
        s = f.read()
        f.close()
        ss = json.loads(s)
        pp.pprint(ss)

        try:
            for num in range(0,len(ss['data'])):
                print '*************************'
                for majorkey, subdict in ss['data'][num].iteritems(): #'paging' or 'data'?
                    print majorkey
                    #print subdict
                    #for subkey, value in subdict.iteritems():
                    #   print subkey, value
                print '  '+str(ss['data'][num]['created_time'].encode('ascii', 'ignore'))
                createdtime = str(ss['data'][num]['created_time'].encode('ascii', 'ignore'))
                print '  '+str(ss['data'][num]['type'].encode('ascii', 'ignore'))+'!'
                fbtype = str(ss['data'][num]['type'].encode('ascii', 'ignore'))
                try:
                    print ' msg '+str(ss['data'][num]['message'].replace("'","''").replace(';','').encode('ascii', 'ignore'))+'!'
                    fbmessage = str(ss['data'][num]['message'].replace("'","''").replace(';','').encode('ascii', 'ignore'))
                except:
                    fbmessage = ''
                try:
                    print ' sty '+str(ss['data'][num]['story'].replace("'","''").replace(';','').encode('ascii', 'ignore'))+'!'
                    fbstory = str(ss['data'][num]['story'].replace("'","''").replace(';','').encode('ascii', 'ignore'))
                except:
                    fbstory = ''

                try:
                    print '  '+str(ss['data'][num]['comments']['count'])+'.'
                    fbcomments = str(ss['data'][num]['comments']['count'])
                except:
                    fbcomments = 0
                try:
                    print '  '+str(ss['data'][num]['likes']['count'])+'.'
                    fblikes = str(ss['data'][num]['likes']['count'])
                except:
                    fblikes = 0
                try:
                    print '  '+str(ss['data'][num]['shares']['count'])+'.'
                    fbshares = str(ss['data'][num]['shares']['count'])
                except:
                    fbshares = 0
                try:
                    print '  '+str(ss['data'][num]['link'])+'.'
                    link = str(ss['data'][num]['link']).replace("'","''")
                except:
                    link = ''
                print '*************************'

                print str(ss['data'][num]['id'])
                fid = str(ss['data'][num]['id'])
                print str(ss['data'][num]['from']['name'].replace("'","''").encode('ascii', 'ignore'))
                from_name = str(ss['data'][num]['from']['name'].replace("'","''").encode('ascii', 'ignore'))
                print str(ss['data'][num]['from']['id'])
                from_id = str(ss['data'][num]['from']['id'])

                query = """insert into fbopengraph.dbo.searches (fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id)
                    values ('"""+str(fid)+"""', '"""+str(from_id)+"""', '"""+str(from_name)+"""', '"""+str(keyword)+"""', '"""+str(fbtype)+"""', '"""+str(link)+"""',convert(datetime,cast(left('"""+str(createdtime)+"""',19) as nvarchar(50)),126), '"""+str(fbmessage)+"""', '"""+str(fbstory)+"""', '"""+str(fblikes)+"""', '"""+str(fbcomments)+"""', '"""+str(fbshares)+"""', getdate(), '"""+str(run_id)+"""')"""
                print query
                cur.execute(query)
                #print '  '+str(ss['data'][num]['message'].encode('ascii', 'ignore'))+' '
                conn.commit()
            #print ss['paging']['next']
            print str(len(ss['data'])) + ' = element "data" length'
            time.sleep(1)

            cur.execute("""insert into searchbank (fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id)
            select distinct fid, from_id, from_name, keyword, type, link, posted, message, story, likes, comments, shares, harvested, run_id from searches where fid NOT in
            (select distinct fid from searchbank)""")

            cur.execute("""delete from searches where keyword = '"""+str(keyword)+"""'""")

            cur.execute("""insert into fbsearchlog (BatchId, keyword, RunDate, HarvestedThisRun, TotalHarvested) values
            (
            '"""+str(run_id)+"""',
            '"""+str(keyword)+"""',
            getdate(),
            ((select count(*) from searchbank where keyword = '"""+str(keyword)+"""')-(select top 1 isnull(TotalHarvested,0) from fbsearchlog where keyword = '"""+str(keyword)+"""' order by RunDate desc)),
            (select count(*) from searchbank where keyword = '"""+str(keyword)+"""')
            )""")

            conn.commit()

            cur.execute("""select top 1 isnull(HarvestedThisRun,TotalHarvested) from fbopengraph.dbo.fbsearchlog where BatchId = '"""+str(run_id)+"""' and keyword = '"""+str(keyword)+"""' order by RUnDate desc""")
            harvienum = cur.fetchone()
            newones = harvienum[0]
            print str(keyword)
            print str(newones) + ' actually kept'

            try:
                nextpage = ss['paging']['next']
                print nextpage
            except:
                nextpage = 'NO'
                print 'No next page avail'
        except:
            print 'Some kind of API errors - invalid data JSON returned maybe?'
            nextpage = 'NO'
            newones = 0
            print 'ERROR'
            print sys.exc_info()[0]
            print sys.exc_info()[1]
            #winsound.PlaySound("*", winsound.SND_ALIAS)




end_script = clock()
print "Finished: %s minutes" % str(end_script / 60)

Again, you only need to change connection info and keyword list.

Like I said, its quick and dirty - but hopefully it's useful to someone!

When using this script - it will initially attempt to search as far back as possible until it runs out of results or hits an error. Upon running it again, it will look for new results and quit digging when it encounters too many dupes (stuff you already have).

All the results will be in the 'searchbank' table, and counts and stats about the data you pulled will be in the 'fbsearchlog' table

Here's some download links the files themselves:

MSSQL Files MYSQL Files

Have fun, any issues - hit me up in the comments!


Related Posts


Tags: , , ,
  • Glenn

    Tried using the following:
    https://graph.facebook.com/search?q=ulrich&type=post&limit=100

    response:

    {
    “error”: {
    “message”: “An unexpected error has occurred. Please retry your request later.”,
    “type”: “OAuthException”,
    “code”: 2
    }
    }

    • http://ryrobes.com/ Ryan Robitaille

      Hmmm, I just tried it and got…

      {
         "data": [
            {
               "id": "1800796607_470354466310956",
               "from": {
                  "name": "Nou00e9mi Kersei",
                  "id": "1800796607"
               },
               "message": "Nos, nem mondhatnu00e1m, hogy szaku00e9rtu0151 vagyok, su0151t, mu00e9g csak utu00e1na sem ju00e1rtam (hogyan is?)... mindenesetre az biztos, hogy rengeteg kaju00e1nak messze nem olyan a minu0151su00e9ge (-> sem u00edze), mint mondjuk gyerekkorunkban (ld. pu00e1rizsi, virsli, gu00e9psonka, kenyu00e9r, tej, paradicsom, u0151szibarack, stb...)",
               "story": "Nou00e9mi Kersei shared Titkolt hu00edrek ku00e9pekben - hogy ku00e9pbe legyu00e9l's photo.",
               "picture": "http://photos-g.ak.fbcdn.net/hphotos-ak-snc7/487890_322095194550928_1881251365_s.jpg",
               "link": "http://www.facebook.com/photo.php?fbid=322095194550928&set=a.283013548459093.63092.167654416661674&type=1",
               "name": "Wall Photos",
               "caption": "A megetetett tu00e1rsadalomrnrnMibu0151l ku00e9szu00fcl az eperfagyi? - Jobb, ha nem tudja! rnNe u00e1mu00edtsuk magunkat! rn- Aki reggelire "kakau00f3italt" iszik, a laboratu00f3rium u00edzeit nyeli. rn- Aki eperjoghurtot kanalaz, fu0171ru00e9szport fal. rn- Aki bolti citromlevet tesz a teu00e1ba, az a penu00e9szgomba "vu00e1ladu00e9ku00e1t" kapja, bu00e1rmennyire tiltakozik is. rn- Az u00e9deset idomu00edtott baktu00e9riumok "ku00f6pik ki", a savanyu00fa a mikroorganizmusok "izzadmu00e1nya".rnu00c9rzu00e9keinket (u00e9szrevu00e9tlenu00fcl) manipulu00e1lja egy vilu00e1ghatalommu00e1 szervezu0151du00f6tt maffia, az u00edz-ipar, s laboratu00f3riumokban elu0151u00e1llu00edtott (titkos) aromu00e1kkal, u00edzfokozu00f3kkal megvu00e1ltoztatja u00edzlelu00e9su00fcnket. Veszu00e9lyben az u00edzlu00e9s szabadsu00e1ga, a kulinu00e1ris u00f6nrendelkezu00e9s joga. Nem azt esszu00fck, amit enni vu00e9lu00fcnk. Mert semmi sem az, aminek lu00e1tszik.rnrnAz eperaromu00e1t egy Holzmindenben szu00e9kelu0151 nu00e9met aromagyu00e1r u00e1llu00edtja elu0151 ausztru00e1l fu0171ru00e9szpor, alkohol, vu00edz u00e9s nu00e9hu00e1ny titkos u00f6sszetevu0151 keveru00e9ku00e9bu0151l. (A Fu00f6ld teljes epertermu00e9se az "eperbu0151l" ku00e9szu00fclt fagylaltok, joghurtok, desszertek mennyisu00e9gu00e9nek egy szu00e1zalu00e9ku00e1t sem fedeznu00e9!) rnrnEgy amerikai konszern baktu00e9riumbu00f3l ku00e1vu00e9tejszu00ednt u00e9s kru00e9msajtot varu00e1zsol, egy mu00e1sik vu00f6ru00f6s u00e1fonyu00e1bu00f3l cseresznyu00e9t, ricinusolajbu00f3l u0151szibarack-aromu00e1t. Az Athlon nevu0171 cu00e9g (nem vicc!) madu00e1rtollbu00f3l pu00e9ksu00fctemu00e9nyt, a General Foods vu00edzbu0151l, fehu00e9rju00e9bu0151l u00e9s gyanu00fas zsiradu00e9kbu00f3l mu0171szalonnu00e1t. rnrnA Procter and Gamble nevu0171 amerikai vu00e1llalat mu00e9g a gyapotszu00e1lat (pamutot) is fogyaszthatu00f3vu00e1 teszi: a nu00f6vu00e9ny rostjaibu00f3l kenyeret su00fctnek. Az u00edzek e szu00e9p u00faj vilu00e1gu00e1ban a menthenthiol nevu0171 vegyu00fclet a gru00e9pfru00fat, az acetyl-pyrolin egyik vu00e1ltozata a ropogu00f3s kenyu00e9rhu00e9j, a filberton a mogyoru00f3u00edz illu00faziu00f3ju00e1u00e9rt felel. rnrnMint a Mu00e1trix cu00edmu0171 filmben. Nem eperjoghurtot eszu00fcnk, hanem vizet mestersu00e9ges illu00faziu00f3kkal. A szagok u00e9s u00edzek e virtuu00e1lis valu00f3su00e1gu00e1ban mu00e1r a fu00fcstu00f6lt sonka u00e9s lazac sem eredeti: napjainkban a hu00fasgyu00e1rak "folyu00e9kony fu00fcstu00f6t", azaz mu0171fu00fcstu00f6t hasznu00e1lnak, amiben 30-90 mu00e1sodperc alatt meg lehet "fu00fcstu00f6lni" a termu00e9ket. Ru00e1adu00e1sul a vu00e1su00e1rlu00f3kat mu00e9zu00e9des hazugsu00e1gokkal etetik. rnrnA Danone Actimel "szuper-egu00e9szsu00e9ges" joghurtital egyszerre tartalmaz ku00e9t (veszu00e9lyesnek tartott) u00e9desu00edtu0151szert, aszpartu00e1mot u00e9s K-aceszulfu00e1mot is. rnrnrnA csalafinta u00edztru00fckku00f6k aztu00e1n megteszik a maguku00e9t, aki gyerekkoru00e1tu00f3l hozzu00e1szokott a vu00edz, szu00e9n-dioxid, karamell, foszforsav, citromsav, nu00e1triumbenzou00e1t, koffeintartalmu00fa italhoz (ez a ku00f3lu00e1k u00f6sszetu00e9tele), annak az igazi bodzaszu00f6rp u00edzetlen. Az u00e9lelmiszeripar tehu00e1t "maszku00edrozza" az u00edzeket, ha kell, korrigu00e1lja u00e9s retusu00e1lja u0151ket (mint Sztu00e1lin ku00f6rnyezetu00e9t a korabeli fotu00f3kon), hozzu00e1szoktat bennu00fcnket a ku00e9miai u00e9desu00edtu0151szerek, u00edzfokozu00f3k, su0171ru00edtu0151anyagok u00edzu00e9hez, reklu00e1mokkal u00e9s aromu00e1kkal etet, u00edzfu00fcggu0151vu00e9 tesz.rnrnKorunkban hu00faszezer u00e9lelmiszer-adalu00e9kot forgalmaznak (ku00f6ztu00fck hu00e9tezer aromu00e1t). E titokzatos porocsku00e1k u00e9s folyadu00e9kok nu00e9lku00fcl, melyek u00f6sszetevu0151it (u00fczleti titokra hivatkozva) sokszor mu00e9g az u00e9telre allergiu00e1s gyerekek orvosainak sem u00e1rulju00e1k el, az u00e9lelmiszer-ipari termu00e9kek tu00f6bbsu00e9ge u00e9lvezhetetlen u00e9s eladhatatlan lenne. *Az u00edz az u00e9telek lelke.* Aki az u00edzeket befolyu00e1solja, u00e9letu00fcnket manipulu00e1lja. Ru00e1adu00e1sul az u00edz nem csak u00edzlu00e9s dolga. Az u00edz legfontosabb funkciu00f3ja a tu00e1plu00e1lu00e9k ellenu0151rzu00e9se. Figyelmeztetu00e9s (lenne), hogy mit nem szabad megenni. Ha becsapju00e1k u00edzlelu00e9su00fcnket, elfeledtetik velu00fcnk az eredeti u00edzt, akkor tu00e9nyleg bu00e1rmit lenyelu00fcnk.rnrnA leves hazudik - u00edrja magyarul u00fajfent megjelent nagyszeru0171 pamfletju00e9ben Hans Ulrich Grimm (Ku00e9tezeregy Kiadu00f3). De nemcsak a Knorr u00e9s a Maggi zacsku00f3s leves hazudik (a nu00e9gy fu0151re szu00f3lu00f3 adagban mindu00f6ssze ku00e9t gramm szu00e1ru00edtott marhahu00fas talu00e1lhatu00f3), hanem szinte minden. rnA kenyu00e9r, a virsli (de az nagyon), a csecsemu0151tu00e1pszer. u00c9rzu00e9keinket manipulu00e1lju00e1k u00e9s u00e1tprogramozzu00e1k. u00c9ppu00fagy, mint gondolatainkat. Egu00e9sz u00e9letu00fcnket. Tavaly az Egyesu00fclt u00c1llamokban az asztalra keru00fclt u00e9teleknek mu00e1r kilencvenhat(!) szu00e1zalu00e9ka ipari termu00e9k volt.rnrnA Fast Food Nation (gyorsu00e9tterem-generu00e1ciu00f3) mu00e1r nem ismeri az u00e9telek eredeti u00f6sszetevu0151it, u00e9letu00e9ben soha nem szagolt vargu00e1nyu00e1t. "De hu00e1t vu00e9gu00fcl is - mondja cinikusan az egyik aromagyu00e1r mu00e9rnu00f6ke - az u00e9lvezet fantu00e1zia dolga." rnMu00e1r akinek!rnrnNe feledd, AZZu00c1 LESZEL AMIT ESZEL! rnKi-ki vonja le a tanulsu00e1got, u00e9s TEGYEN MEG MINDENT A CSALu00c1DJA EGu00c9SZSu00c9Gu00c9u00c9RT!rnrnhttp://biovetomag.hupont.hu/",....

      Let me check it out.

      • http://ryrobes.com/ Ryan Robitaille

        Sorry, that was a horrible cut-n-paste. I should have just said – that I got data. ;)

        Still, I’ll try and replicate your error.

        Did it work before, at all? Those URLs ( https://graph.facebook.com/search?q=ulrich&type=post&limit=100 ) seem to pull results from whatever browser / computer I use..

        • Howie

          {
          “error”: {
          “message”: “(#200) Permissions error”,
          “type”: “OAuthException”,
          “code”: 200
          }
          }

  • http://twitter.com/vazsingh Varinder Singh

    Dare I ask if a PHP version is availible? :-P

    • http://ryrobes.com/ Ryan Robitaille

      Hey Varinder, It’s def do-able. The Python one is pretty easy to replicate though… but let me crank out a PHP for the hell of it. ;)

      • http://twitter.com/vazsingh Varinder Singh

        Hi Ryan, how did you get on with this? Any new developments? :-)

  • Dirk

    For some reason search isn’t working anymore. Is this something you experienced also? I get no results with the facebook api graph

    • Dirk

      Seems to be a facebook error with the graph api. search on posts returns no data.

  • dorkboy

    Does this still work?

    I am trying with the original search terms. The only thing I have changed is my connection string. I get a row written to the fbsearchlog table but no data in the other tables and a “0 actually kept” message.

  • Jojo

    Great stuff!

  • Mike

    Im about to put my cut and paste skills to the test!

  • Philip Hoyos

    Im curious. Does this still work? I have researched the way facebook graph works, and as far as I can tell the service no longer works. I tested it and i get a error 200, something about an invalid access token (Which is very valid if I use the regular facebook insight interface).

    As im still a novice in this field, what would be the steps to start using your code? Something like; 1. Setup DB Server, 2. Configure files, 3. Click .py file?

    Thank you for being a great inspiration! You have a lot of useful stuff that I would like to learn how to do!

  • hugo koopmans

    why use a relational database if you handle json objects? use document store like MarkLogic or BaseX

  • vks2106

    $ sudo python fetch_face_mysql.py

    BatchRun #1

    Some kind of API errors – invalid data JSON returned maybe?

    ERROR

    ‘data’

    Some kind of API errors – invalid data JSON returned maybe?

    This is all am getting and mysql tables are empty as well.

  • Brice Vallieres

    As of v.2.0, they no longer allow public search of fb posts:

    Public post search is no longer available. (/search?type=post&q=foobar)

    https://developers.facebook.com/docs/apps/upgrading

    damn shame…

  • Krete

    Does this work now also?

    • Krete

      Just saw previous post :)