How to Scrape Emails from GitHub
Scraping emails from GitHub can be a useful way to find collaborators, track project progress, or simply get in touch with developers. However, it's essential to respect privacy and GitHub's terms of service. This guide will walk you through the process responsibly.
Before we begin, ensure you understand that scraping should be done judiciously and ethically. Always anonymize or delete data after use, and never send unsolicited messages. Also, respect GitHub's Terms of Service to avoid any legal issues.
Understanding GitHub's Data Export API
GitHub provides a data export API that allows users to download their data in JSON format. While this doesn't directly give us emails, it's a starting point. The ' followers' field in the JSON response contains usernames, from which we can extract email addresses.
To access this API, you'll need to authenticate with GitHub using an access token. You can generate one from your personal access tokens page.
Setting Up Your Environment
You'll need Python with the requests library to interact with GitHub's API. Install it using pip: pip install requests.
Here's a simple script to fetch your followers using the API:
```python import requests import json headers = {'Authorization': 'token YOUR_ACCESS_TOKEN'} response = requests.get('https://api.github.com/users/USERNAME/followers', headers=headers) followers = json.loads(response.text) ```Extracting Email Addresses
Emails are not immediately visible in the response. They're in the 'email' field, but it's empty for many users due to privacy settings. We can find them by searching for the username in the format 'username@users.noreply.github.com'.
Here's how you can do this for a list of followers:
```python for follower in followers: username = follower['login'] url = f'https://api.github.com/users/{username}' response = requests.get(url, headers=headers) data = json.loads(response.text) if 'email' in data: print(f"{username}: {data['email']}") else: print(f"{username}: Not public") ```Scraping GitHub Pages
Some users might not make their emails public on their profile. In such cases, you can check their GitHub pages, which often display contact information. However, be aware that scraping personal websites without permission is invasions of privacy.
Only scrape public GitHub pages (/username.github.io) for this purpose, and do so responsibly.
Accessing GitHub Pages
To access a user's GitHub pages, you can use the following URL format: https://username.github.io/.
Here's a simple script to fetch the content of a GitHub page:
```python import requests url = 'https://username.github.io/' response = requests.get(url) content = response.text ```Parsing HTML for Email Addresses
Once you have the content, you can parse the HTML to find email addresses. Python's beautifulsoup4 library is excellent for this. Install it using pip: pip install beautifulsoup4.
Here's how you can find email addresses in the content:
```python from bs4 import BeautifulSoup soup = BeautifulSoup(content, 'html.parser') emails = soup.find_all('a', href=True) for email in emails: if 'mailto:' in email['href']: print(email['href'].replace('mailto:', '')) ```