Set up your own Google Custom Search Engine
A while ago, I was trying to integrate a recipe search functionality into my grocery shopping dashboard. To get Google search results programmatically, I found this code snippet by the Stack Overflow user mbdevpl using the Python client library for Google’s discovery based APIs:
## Import libraries
from googleapiclient.discovery import build
## Set credentials
my_api_key = "API_key"
my_cse_id = "CSE_ID"
## Define function
def google_search(search_term, api_key, cse_id, **kwargs):
service = build("customsearch", "v1", developerKey=api_key)
res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
return res['items']mbdevpl provided very detailed instructions for setting up and querying the Google Custom Search Engine (be sure to follow all the steps in the answer). Creating your own Google custom search engine and Google API key is free.
As an example, if I want to look up a recipe that can uses up the peaches, turmeric, milk and spinach in my fridge:
## Generate search terms
ing_list = ['peaches', 'turmeric', 'milk', 'spinach']
search_term = '+'.join(ing_list)
## Make query, limit to 10 results
results = google_search(search_term, my_api_key, my_cse_id, num=10)The search results are returned as a list of dictionaries. We can quickly look at one of them:
from pprint import pprint
pprint(results[2], indent=2)## { 'cacheId': '5KT9gF2p3SgJ',
## 'displayLink': 'blog.fitbit.com',
## 'formattedUrl': 'https://blog.fitbit.com/the-ultimate-green-smoothie-recipe/',
## 'htmlFormattedUrl': 'https://blog.fitbit.com/the-ultimate-green-smoothie-recipe/',
## 'htmlSnippet': 'Mar 17, 2016 <b>...</b> Dark greens: <b>spinach</b>, kale, '
## 'chard. Fruit: banana, <b>peaches</b>, mango, blueberries, '
## '<br>\n'
## 'avocado ... Flavor: <b>ginger</b>, <b>turmeric</b>, matcha, '
## 'parsley, cilantro ... In a blender, add <br>\n'
## 'the kale and almond <b>milk</b> and blend to break down the '
## 'leaves.',
## 'htmlTitle': 'The Ultimate Green Smoothie Recipe - Fitbit Blog',
## 'kind': 'customsearch#result',
## 'link': 'https://blog.fitbit.com/the-ultimate-green-smoothie-recipe/',
## 'pagemap': { 'cse_image': [ { 'src': 'https://blog.fitbit.com/wp-content/uploads/2016/03/2016-12-16_Ultimate_Green_Smoothie_Blog_730x485.jpg'}],
## 'cse_thumbnail': [ { 'height': '183',
## 'src': 'https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcTqkQPP0AwKwTlMhVvPZ6maabGBMx_7Mh763Mab8SKa8q4EGYeKUzja96U',
## 'width': '276'}],
## 'hcard': [ {'fn': 'Merrilyn', 'nickname': 'Merrilyn'},
## {'fn': 'Anonymous', 'nickname': 'Anonymous'}],
## 'metatags': [ { 'article:modified_time': '2019-02-01T20:04:05+00:00',
## 'article:published_time': '2016-03-17T15:00:35+00:00',
## 'article:publisher': 'https://facebook.com/fitbit',
## 'article:section': 'Eat Well',
## 'article:tag': 'recipes',
## 'msapplication-tilecolor': '#ffffff',
## 'msapplication-tileimage': 'https://blog.fitbit.com/wp-content/themes/fitbit-new/assets/img/icons/favicon//ms-icon-144x144.png',
## 'og:description': 'A big green drink is an easy '
## 'and delicious way to slip '
## 'superfoods into your day. '
## 'Make the perfect smoothie, '
## 'featuring veggies, protein, '
## 'and healthy fats.',
## 'og:image': 'https://blog.fitbit.com/wp-content/uploads/2016/03/2016-12-16_Ultimate_Green_Smoothie_Blog_730x485.jpg',
## 'og:image:height': '485',
## 'og:image:secure_url': 'https://blog.fitbit.com/wp-content/uploads/2016/03/2016-12-16_Ultimate_Green_Smoothie_Blog_730x485.jpg',
## 'og:image:width': '730',
## 'og:locale': 'en_US',
## 'og:site_name': 'Fitbit Blog',
## 'og:title': 'The Ultimate Green Smoothie Recipe '
## '- Fitbit Blog',
## 'og:type': 'article',
## 'og:updated_time': '2019-02-01T20:04:05+00:00',
## 'og:url': 'https://blog.fitbit.com/the-ultimate-green-smoothie-recipe/',
## 'theme-color': '#ffffff',
## 'twitter:card': 'summary',
## 'twitter:creator': '@fitbit',
## 'twitter:description': 'A big green drink is an '
## 'easy and delicious way '
## 'to slip superfoods into '
## 'your day. Make the '
## 'perfect smoothie, '
## 'featuring veggies, '
## 'protein, and healthy '
## 'fats.',
## 'twitter:image': 'https://blog.fitbit.com/wp-content/uploads/2016/03/2016-12-16_Ultimate_Green_Smoothie_Blog_730x485.jpg',
## 'twitter:site': '@fitbit',
## 'twitter:title': 'The Ultimate Green Smoothie '
## 'Recipe - Fitbit Blog',
## 'viewport': 'user-scalable=yes, '
## 'width=device-width, '
## 'initial-scale=1, '
## 'maximum-scale=1'}]},
## 'snippet': 'Mar 17, 2016 ... Dark greens: spinach, kale, chard. Fruit: '
## 'banana, peaches, mango, blueberries, \n'
## 'avocado ... Flavor: ginger, turmeric, matcha, parsley, cilantro '
## '... In a blender, add \n'
## 'the kale and almond milk and blend to break down the leaves.',
## 'title': 'The Ultimate Green Smoothie Recipe - Fitbit Blog'}Here is how to grab bits of information that I want from each dictionary and put them into a pandas dataframe:
## Import library
import pandas as pd
## Create empty dataframe with desired columns
df = pd.DataFrame(columns = ['Title', 'URL', 'Description', 'Image'])
## Append results to dataframe row by row
for result in results:
df = df.append({'Title':result['title'],
'Description':result['snippet'],
'Image':result['pagemap']['cse_image'][0]['src'],
'URL':result['formattedUrl']},
ignore_index=True)As a quick demonstration, the R package kableExtra can produce very sleek tables in which the image URLs can be rendered as images. Of course, much more styling can be done to prepare the content for an actual application.
## Import libraries
library(dplyr, quietly = T)
library(knitr, quietly = T)
library(kableExtra)
## Import dataframe variable from Python runtime
df <- py$df
## Preprocess the image links to allow rendering
df$Image = sprintf('', df$Image)
## Hyperlink the titles
df$Title = sprintf("[%s](%s)", df$Title, df$URL)
## Drop the URL column
df$URL <- NULL
kable(df) %>%
kable_styling(full_width = F) %>%
column_spec(1, bold = T) %>%
column_spec(2, width = "20em")| Title | Description | Image |
|---|---|---|
| Mango Ginger Kale Green Smoothie | Minimalist Baker Recipes | I used spinach instead of kale..what I had on hand. Organic mango juice (from Costco) as the base, spinach, frozen peaches, fresh grated ginger as specified, … |
|
| Peach Green Smoothie Recipe - Pinch of Yum | Jul 19, 2013 … This simple peach green smoothie has frozen and fresh peaches, honey, ginger, and spinach. Super healthy and so refreshing. |
|
| The Ultimate Green Smoothie Recipe - Fitbit Blog | Mar 17, 2016 … Dark greens: spinach, kale, chard. Fruit: banana, peaches, mango, blueberries, avocado … Flavor: ginger, turmeric, matcha, parsley, cilantro … In a blender, add the kale and almond milk and blend to break down the leaves. |
|
| The only green smoothie you’ll ever need! Fresh spinach and flax … | Fresh spinach and flax seeds are sweetened with citrus and bananas. … ginger turmeric lemonade recipe made with Whole Foods: fresh ginger and turmeric root and …. is made with banana, frozen mango, baby spinach, and almond milk. |
|
| Ginger Peach & Kale Power Smoothie | Aug 24, 2015 … Ginger, peaches and kale make a nutrient-dense green power … I love myself a delicious green smoothie and I always make it a goal to get spinach, kale or … If coconut milk isn’t your thing, you can substitute for almond or … |
|
| Summer Spinach, Peach and Pecan Salad recipe – All recipes … | A perfect summer salad with ripe peaches, spinach and toasted pecans. Perfect as a side dish at barbecues. |
|
| Banana, Avocado, and Spinach Smoothie Recipe - Allrecipes.com | A banana, avocado, and spinach smoothie is a quick and easy breakfast or snack … Blend banana, avocado, spinach, milk, ice cubes, honey, and vanilla extract … |
|
| Peach - All recipes Australia NZ | This is a super easy dessert of peaches drizzled with honey and cream cheese. … A perfect summer salad with ripe peaches, spinach and toasted pecans. |
|
| 10 Best Peach Pear Smoothie Recipes | 5 days ago … kale, ginger, ice, almond milk, banana, peaches, matcha … water, protein powder , flax seeds, almond butter, spinach, pear and 7 more. |
|
| Creamy Spinach Soup With Golden Quinoa | MyFitnessPal | Jan 1, 2019 … This non-dairy version of creamed spinach soup gets its creamy texture from miso and almond milk. The turmeric-tinted quinoa lends some … |
|
The data obtained can also be used for much more, like text processing and image recognition.
Have fun with it! :)