-
-
Notifications
You must be signed in to change notification settings - Fork 372
Added Custom-search CLI script #485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a new Custom-search CLI tool that allows users to perform Google Custom Search API queries and export results to CSV format. The implementation includes a Python script with command-line interface for searching, API key management, pagination support, and CSV output functionality.
Key Changes
- Added a complete CLI tool for Google Custom Search API integration with settings management and CSV export
- Created supporting documentation and configuration files
- Updated the main README to include the new tool in the project listing
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
File | Description |
---|---|
README.md | Added entry for the new Custom-search CLI tool in the project table |
Custom-search CLI/scraper.py | Main Python script implementing the search functionality, API management, and CSV export |
Custom-search CLI/setting.json | Configuration file template for API credentials |
Custom-search CLI/readme.md | Documentation for the CLI tool with setup instructions and usage examples |
Custom-search CLI/output.csv | Empty CSV template file for search results |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| CSV to Excel | [CSV to Excel](https://github.com/DhanushNehru/Python-Scripts/tree/main/CSV%20to%20Excel) | A Python script to convert a CSV to an Excel file. | | ||
| CSV_TO_NDJSON | [CSV to Excel](https://github.com/DhanushNehru/Python-Scripts/tree/main/CSV_TO_NDJSON) | A Python script to convert a CSV to an NDJSON files file. | | ||
| Currency Script | [Currency Script](https://github.com/DhanushNehru/Python-Scripts/tree/main/Currency%20Script) | A Python script to convert the currency of one country to that of another. | | ||
| Custom-search CLI | [Custom-search CLI](https://github.com/DhanushNehru/Python-Scripts/tree/main/Custom-search%20CLI) | Python script to search a query through internet and save the results in a .csv file. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Title contains a spelling error: 'Cutom-search' should be 'Custom-search'. Also, there are excessive spaces between 'Custom-search' and 'CLI' that should be normalized.
| Custom-search CLI | [Custom-search CLI](https://github.com/DhanushNehru/Python-Scripts/tree/main/Custom-search%20CLI) | Python script to search a query through internet and save the results in a .csv file. | | |
| Custom-search CLI | [Custom-search CLI](https://github.com/DhanushNehru/Python-Scripts/tree/main/Custom-search%20CLI) | Python script to search a query through internet and save the results in a .csv file. | |
Copilot uses AI. Check for mistakes.
pip install beautifulsoup4 | ||
pip install python-csv | ||
pip install argparse |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The dependencies listed are incorrect. The script doesn't use beautifulsoup4 or python-csv (csv is built-in), and argparse is part of the standard library. Only 'requests' is needed as an external dependency.
pip install beautifulsoup4 | |
pip install python-csv | |
pip install argparse |
Copilot uses AI. Check for mistakes.
2. Create a Custom Search Engine (CX) at [Google CSE](https://cse.google.com/cse/all) | ||
3. Run the script with your API key to create setting.json: | ||
|
||
python main.py -sq [SEARCH_QUERY] --add_api_key [YOUR_API_KEY] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The filename in the example is incorrect. It should be 'scraper.py' instead of 'main.py' to match the actual script filename.
python main.py -sq [SEARCH_QUERY] --add_api_key [YOUR_API_KEY] | |
python scraper.py -sq [SEARCH_QUERY] --add_api_key [YOUR_API_KEY] |
Copilot uses AI. Check for mistakes.
api_key = settings["API_KEY"] | ||
cx = settings["CX"] | ||
|
||
print(f"Using API key: {api_key}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Printing the API key to console poses a security risk as it could expose sensitive credentials in logs or terminal history. Consider printing only a masked version or removing this line entirely.
print(f"Using API key: {api_key}") | |
print(f"Using API key: {api_key[:4]}...{api_key[-4:]} (masked)") |
Copilot uses AI. Check for mistakes.
PR Title
Custom-search CLI script (#460)
Summary
Implemented a Python script that uses Google Custom Search API to perform queries and export results to CSV.
Description
This script implements a Google Custom Search scraper with the following features:
setting.json
(create the file if missing and allows adding the api key with--add_api_key
)--search_query
or-sq
)--page
commandoutput.csv
fileThe changes are as follows:
scraper.py
(main python script)setting.json
to fetch API keyoutput.csv
for saving resultsargparse
supporting options for search query, pagination and API key managementChecks
in the repository
in the PR
Thank You,
Javad-dd