How to Build an Earnings Call Transcript Analyzer

Earnings calls contain signals that move stock prices — hedging language, forward guidance changes, and sentiment shifts. Most investors read transcripts manually. We’ll build a Python tool that scrapes transcripts and extracts actionable signals automatically.

The Value of Automated Transcript Analysis

When a CEO shifts from “we expect strong growth” to “we anticipate headwinds,” that language change predicts stock movement. NLP can detect these shifts across hundreds of companies simultaneously — something no human analyst can do.

Scraping Earnings Transcripts

Several sites publish free earnings call transcripts. We’ll build a scraper that collects them systematically using ScraperAPI for reliable proxy rotation:

import requests
from bs4 import BeautifulSoup
import re

SCRAPER_API_KEY = "YOUR_KEY"

def scrape_transcript(url):
    response = requests.get(
        "http://api.scraperapi.com",
        params={
            "api_key": SCRAPER_API_KEY,
            "url": url,
            "render": "true"
        },
        timeout=60
    )
    soup = BeautifulSoup(response.text, "html.parser")
    segments = []
    current_speaker = None
    current_text = []
    current_role = ""

    for element in soup.find_all(["p", "strong", "b"]):
        text = element.get_text(strip=True)
        speaker_match = re.match(r"^([A-Z][a-z]+ [A-Z][a-z]+)s*[-u2014]s*(.+)", text)
        if speaker_match:
            if current_speaker:
                segments.append({
                    "speaker": current_speaker,
                    "role": current_role,
                    "text": " ".join(current_text)
                })
            current_speaker = speaker_match.group(1)
            current_role = speaker_match.group(2)
            current_text = []
        elif current_speaker:
            current_text.append(text)
    return segments

Sentiment Analysis Engine

We analyze sentiment at the sentence level to catch nuanced shifts:

from collections import Counter
import re

POSITIVE_TERMS = {
    "growth", "exceeded", "strong", "record", "confident",
    "momentum", "opportunity", "upside", "robust", "accelerating",
    "outperformed", "beat", "raised", "guidance", "optimistic"
}

NEGATIVE_TERMS = {
    "headwinds", "challenging", "decline", "pressure", "uncertainty",
    "cautious", "softness", "deceleration", "missed", "below",
    "restructuring", "impairment", "risk", "concern", "weakness"
}

HEDGING_TERMS = {
    "may", "might", "could", "potentially", "possibly",
    "approximately", "somewhat", "relatively", "generally"
}

def analyze_segment(segment):
    words = set(re.findall(r'bw+b', segment["text"].lower()))
    pos_count = len(words & POSITIVE_TERMS)
    neg_count = len(words & NEGATIVE_TERMS)
    hedge_count = len(words & HEDGING_TERMS)
    total = pos_count + neg_count or 1
    sentiment_score = (pos_count - neg_count) / total

    return {
        "speaker": segment["speaker"],
        "role": segment["role"],
        "sentiment": sentiment_score,
        "hedging_ratio": hedge_count / max(len(words), 1),
        "positive_terms": list(words & POSITIVE_TERMS),
        "negative_terms": list(words & NEGATIVE_TERMS),
        "word_count": len(words)
    }

Forward Guidance Detection

Extract and compare forward-looking statements:

def extract_guidance(segments):
    guidance_patterns = [
        r"(?:we expect|we anticipate|we project|we forecast|guidance).{10,200}",
        r"(?:next quarter|full year|fiscal year).{10,200}",
        r"(?:revenue.{0,30}(?:between|range|expect)|eps.{0,30}(?:between|range|expect)).{10,100}",
    ]
    guidance_statements = []
    for segment in segments:
        if segment.get("role", "").lower() in ["ceo", "cfo", "chief executive", "chief financial"]:
            for pattern in guidance_patterns:
                matches = re.findall(pattern, segment["text"].lower())
                for match in matches:
                    guidance_statements.append({
                        "speaker": segment["speaker"],
                        "role": segment["role"],
                        "statement": match.strip(),
                        "type": "forward_guidance"
                    })
    return guidance_statements

Quarter-over-Quarter Comparison

The real value emerges when you compare transcripts across quarters:

def compare_quarters(current_analysis, previous_analysis):
    changes = {}
    for speaker in set(a["speaker"] for a in current_analysis):
        curr = [a for a in current_analysis if a["speaker"] == speaker]
        prev = [a for a in previous_analysis if a["speaker"] == speaker]
        if curr and prev:
            sentiment_shift = curr[0]["sentiment"] - prev[0]["sentiment"]
            hedging_shift = curr[0]["hedging_ratio"] - prev[0]["hedging_ratio"]
            changes[speaker] = {
                "sentiment_change": round(sentiment_shift, 3),
                "hedging_change": round(hedging_shift, 3),
                "signal": (
                    "BEARISH" if sentiment_shift < -0.2 or hedging_shift > 0.05
                    else "BULLISH" if sentiment_shift > 0.2
                    else "NEUTRAL"
                )
            }
    return changes

Scaling the Analysis

To analyze hundreds of earnings calls per quarter, you need robust proxy infrastructure. ThorData provides residential proxies for scraping financial sites, and ScrapeOps monitors scraper performance across sources.

Putting It All Together

def full_analysis_pipeline(transcript_url):
    segments = scrape_transcript(transcript_url)
    analysis = [analyze_segment(s) for s in segments]
    guidance = extract_guidance(segments)
    ceo_sentiment = [a for a in analysis if "ceo" in a["role"].lower()]
    overall = sum(a["sentiment"] for a in analysis) / len(analysis) if analysis else 0
    return {
        "overall_sentiment": round(overall, 3),
        "ceo_sentiment": ceo_sentiment,
        "guidance_statements": guidance,
        "segment_count": len(segments)
    }

Key Insights

• Hedging increases in the Q&A section often predict earnings misses next quarter
• CEO vs CFO sentiment divergence can signal internal disagreements about outlook
• Word count changes (shorter answers) sometimes indicate discomfort with a topic

Build this analyzer, run it on historical transcripts, and backtest against stock performance. The patterns are real and exploitable.

Let me describe a scene every LaTeX user has lived:

It’s 11pm. You’ve been writing all day. You hit compile, walk away to make tea, come back to find this in the log:

! Package inputenc Error: Unicode character 的 (U+7684)
(inputenc)                not set up for use with LaTeX.

See the inputenc package documentation for explanation.
Type  H <return>  for immediate help.
 ...
l.1847 ...rithm based on the work of Li et al. 的

?

And you just stare at it. You didn’t even type a Chinese character. Where did that come from? Is it in a bibliography entry? A comment someone else left? A package that auto-inserted something?

This is the specific kind of LaTeX suffering that has no good solution — until recently.

The problem with current LaTeX debugging tools

When your compile log hits 400 lines, the challenge isn’t understanding individual error messages. It’s triage — figuring out which errors are root causes and which ones are just cascading side effects of a single upstream problem.

I used to do this manually. Open the log. Search for !. Copy the error. Google it. Read StackOverflow answers that almost match my situation. Try a fix. Recompile. Different error. Repeat.

Calling ChatGPT helps a little. But here’s the issue: ChatGPT doesn’t know your codebase. You paste a snippet, it gives you a generic answer. It doesn’t know what packages you’ve loaded, what custom commands you’ve defined, or what your root file structure looks like. So you get answers like “make sure amsmath is loaded” when you already have amsmath loaded and the real problem is something three files upstream.

What TeX64’s Axiom does differently

TeX64 is a local-first LaTeX editor for macOS. Its AI assistant, Axiom, has access to your entire project — all .tex files, your actual compile log, and your BibTeX. Not a snippet you pasted. The whole thing.

This changes the quality of the answers dramatically.

Here’s a real example. I had a project with 23 errors in the log. I typed “analyze these errors” into Axiom’s chat. The response:

“You have 23 errors but they stem from 3 root causes:

Root cause 1 (generates 18 errors): macros.tex line 34 defines proof using newcommand, but the amsthm package already defines a proof environment. This causes a naming conflict that cascades throughout the document.

Root cause 2 (generates 4 errors): chapter4.tex line 209 has a begin{table} with no matching end{table} — LaTeX loses track of the environment and the errors pile up from there.

Root cause 3 (1 error): bibliography.bib entry Zhang2019 has a { in the title field that’s not closed.

I’ll show diffs for each fix. Want to start with root cause 1?”

That analysis would have taken me an hour of manual log reading. It took Axiom about 20 seconds.

The diff-before-apply approach

One thing Axiom does that I deeply appreciate: it shows you exactly what it wants to change before changing anything.

Every suggestion comes with a diff:

  % macros.tex, line 34
- newcommand{proof}[1]{textit{Proof.} #1 qed}
+ newcommand{myproof}[1]{textit{Proof.} #1 qed}

And then: “Apply this change?”

You see what’s happening, you understand why, and you decide. Axiom never silently edits your files. For a thesis or paper where every word matters, this is the right way to work with AI.

Other things I’ve used it for

Once I started using TeX64, I found Axiom useful for things beyond error fixing:

Generating math I don’t remember the syntax for: I can describe a formula in plain English — “the limit as n approaches infinity of the sum from k=1 to n of 1/k^2” — and get clean LaTeX back. No more digging through the LaTeX math symbol list.

Literature: paste title, get BibTeX: Tell Axiom you want to cite a paper, it searches arXiv and drops the BibTeX entry into your .bib file. This alone has saved me probably 20 minutes per writing session.

Package conflict diagnosis: When adding packages to a big project causes mysterious new errors, Axiom can spot ordering conflicts. “You’re loading cleveref before hyperref — switch the order.”

Honest limitations

TeX64 is macOS only. If you’re on Windows or Linux, this isn’t for you.

You need MacTeX or TeX Live installed separately — TeX64 is the editor, not a LaTeX distribution. The app’s environment diagnostics will walk you through setup if you don’t have it already.

AI features require internet. Compilation itself is fully local and offline-capable. On my M2 MacBook Air, a 120-page document compiles in about 8 seconds — no internet bottleneck on the compile step.

There’s a free tier to start with. If you end up using Axiom heavily, there are paid plans.

The thing that surprised me most

I expected Axiom to be useful. What I didn’t expect was how much it changed my relationship with errors.

Before: compile error → dread → 45-minute debugging session.

After: compile error → “let Axiom look at this” → usually fixed in under 5 minutes.

The psychological difference is real. When you’re not scared of errors, you compile more often. When you compile more often, you catch problems earlier. It’s a virtuous loop that actually makes you write more.

If you’re on a Mac and spend meaningful time in LaTeX, it’s worth trying.

→ tex64.com

Google’s Cache Compression, Siri’s Open Door, and the CFO Agent Test

Google slashes AI memory needs by 6x while Apple opens Siri to rivals, and a new AI-focused language arrives as LLM agents face their first CFO benchmark.

Google’s TurboQuant reduces AI LLM cache memory capacity requirements by at least six times — up to 8x performance boost on Nvidia H100 GPUs, compresses KV caches to 3 bits with no accuracy loss – Tom

What happened:

Google’s TurboQuant cuts LLM cache memory requirements by at least six times, achieving up to 8x performance gains on Nvidia H100 GPUs by compressing KV caches to just 3 bits without accuracy loss.

Why it matters:

This breakthrough directly addresses the memory bottleneck that limits LLM inference speed and scale, potentially enabling larger models to run on existing hardware or dramatically reducing infrastructure costs for AI services.

Context:

KV cache compression has been a key focus area as models grow larger and inference costs become prohibitive for many applications.

Apple Plans to Open Up Siri to Rival AI Assistants in iOS 27 Update

What happened:

Apple plans to allow Siri to integrate with competing AI assistants beyond ChatGPT in the upcoming iOS 27 update, according to Bloomberg.

Why it matters:

This marks a significant shift from Apple’s traditionally closed ecosystem, potentially giving developers and users more flexibility while forcing Apple to compete on AI quality rather than lock-in.

Context:

The move comes as Apple faces pressure to keep pace with rapidly advancing AI capabilities from competitors like OpenAI and Google.

Aria – A programming language specifically for AI code generation

What happened:

Aria is a new programming language designed specifically for AI code generation, with its website and documentation now available online.

Why it matters:

Purpose-built languages for AI development could streamline workflows and reduce the friction between human intent and machine-generated code, potentially becoming a standard tool for AI-assisted programming.

Context:

As AI coding tools mature, specialized languages may emerge to bridge the gap between natural language prompts and executable code.

OpenAI Backs Isara, New AI Startup Seeking Bot Army Breakthroughs

What happened:

OpenAI has backed Isara, a new AI startup focused on developing breakthroughs for bot army technology, according to the Wall Street Journal.

Why it matters:

This investment signals OpenAI’s interest in scaling AI agents beyond single-task assistants toward coordinated multi-agent systems, which could transform everything from customer service to software testing.

Context:

The “bot army” concept represents the next frontier in agentic AI, where multiple specialized agents work together autonomously.

Software Engineer Interviews for the Age of AI

What happened:

A blog post explores how software engineer interviews should evolve in the age of AI, with discussion on Hacker News.

Why it matters:

As AI tools become ubiquitous in development, traditional coding interviews may need to test problem-solving and system design skills rather than memorization or manual coding speed.

Context:

Companies are grappling with how to evaluate engineers when AI can handle much of the routine coding work.

Can LLM Agents Be CFOs? A Benchmark for Resource Allocation in Dynamic Enterprise Environments

What happened:

Researchers have created a benchmark testing whether LLM agents can handle CFO-level resource allocation decisions in dynamic enterprise environments, addressing uncertainty and competing objectives.

Why it matters:

This benchmark tackles one of AI’s biggest challenges: making strategic, long-term decisions with incomplete information—a capability that could transform business operations if achieved.

Context:

While LLMs excel at reactive tasks, complex resource allocation requires planning and trade-offs that remain difficult for current AI systems.

Sources: Google News AI, Hacker News AI, Arxiv AI

Building a Real Estate Foreclosure Tracker with Public Records

Foreclosure data is public information in the United States, but it’s scattered across county clerk websites, court systems, and government databases. Investors, researchers, and journalists who can aggregate this data gain a significant informational advantage. Let’s build a Python-based foreclosure tracker.

Why Track Foreclosures?

Foreclosure filings are leading indicators of economic stress in specific markets. They help real estate investors find below-market properties, help journalists investigate predatory lending, and help researchers study housing market dynamics.

Data Sources

Foreclosure data comes from several public sources:

• County clerk/recorder websites — Notice of Default, Lis Pendens filings
• HUD foreclosure listings — government-owned properties
• Court records — judicial foreclosure proceedings
• Census/ACS data — demographic context

HUD Foreclosure Listings Scraper

import requests
from bs4 import BeautifulSoup
import re

SCRAPER_API_KEY = "YOUR_KEY"

class HUDForeclosureScraper:
    BASE_URL = "https://www.hudhomestore.gov"

    def search_properties(self, state, city=None, zip_code=None):
        search_url = f"{self.BASE_URL}/Listing/PropertySearchResult"
        form_data = {
            "State": state,
            "City": city or "",
            "Zip": zip_code or "",
            "PropertyType": "ALL",
            "ListingStatus": "Available"
        }
        response = requests.post(
            "http://api.scraperapi.com",
            params={"api_key": SCRAPER_API_KEY, "url": search_url, "render": "true"},
            data=form_data,
            timeout=60
        )
        return self._parse_listings(response.text)

    def _parse_listings(self, html):
        soup = BeautifulSoup(html, "html.parser")
        properties = []
        for listing in soup.select(".property-listing, .listing-row, tr"):
            address = listing.select_one(".address, td:nth-child(1)")
            price = listing.select_one(".price, td:nth-child(3)")
            status = listing.select_one(".status, td:nth-child(4)")
            if address:
                properties.append({
                    "address": address.get_text(strip=True),
                    "price": price.get_text(strip=True) if price else "N/A",
                    "status": status.get_text(strip=True) if status else "N/A"
                })
        return properties

County Records Scraper

class CountyRecordsScraper:
    def __init__(self, scraper_api_key):
        self.api_key = scraper_api_key

    def scrape_county_records(self, county_url, search_params):
        response = requests.get(
            "http://api.scraperapi.com",
            params={"api_key": self.api_key, "url": county_url, "render": "true"},
            timeout=60
        )
        soup = BeautifulSoup(response.text, "html.parser")
        filings = []
        tables = soup.find_all("table")
        for table in tables:
            rows = table.find_all("tr")
            for row in rows[1:]:
                cells = row.find_all("td")
                if len(cells) >= 4:
                    filing = {
                        "case_number": cells[0].get_text(strip=True),
                        "filing_date": cells[1].get_text(strip=True),
                        "document_type": cells[2].get_text(strip=True),
                        "property_address": cells[3].get_text(strip=True)
                    }
                    foreclosure_types = [
                        "notice of default", "lis pendens",
                        "notice of sale", "foreclosure"
                    ]
                    if any(ft in filing["document_type"].lower() for ft in foreclosure_types):
                        filings.append(filing)
        return filings

Geocoding and Market Analysis

import time

def geocode_address(address):
    response = requests.get(
        "https://nominatim.openstreetmap.org/search",
        params={"q": address, "format": "json", "limit": 1},
        headers={"User-Agent": "ForeclosureTracker/1.0"}
    )
    time.sleep(1)
    results = response.json()
    if results:
        return {
            "lat": float(results[0]["lat"]),
            "lon": float(results[0]["lon"]),
            "display_name": results[0]["display_name"]
        }
    return None

def market_concentration(properties):
    from collections import Counter
    zips = []
    for prop in properties:
        zip_match = re.search(r'b(d{5})b', prop.get("address", ""))
        if zip_match:
            zips.append(zip_match.group(1))
    concentration = Counter(zips).most_common(20)
    return [{"zip": z, "count": c} for z, c in concentration]

Building the Pipeline

import pandas as pd
from datetime import datetime

class ForeclosureTracker:
    def __init__(self, api_key):
        self.hud = HUDForeclosureScraper()
        self.county = CountyRecordsScraper(api_key)

    def daily_scan(self, states, county_urls):
        all_properties = []
        for state in states:
            properties = self.hud.search_properties(state)
            for p in properties:
                p["source"] = "HUD"
                p["state"] = state
            all_properties.extend(properties)
        for url in county_urls:
            filings = self.county.scrape_county_records(url, {})
            for f in filings:
                f["source"] = "County"
            all_properties.extend(filings)
        df = pd.DataFrame(all_properties)
        df["scan_date"] = datetime.now().isoformat()
        filename = f"foreclosures_{datetime.now():%Y%m%d}.csv"
        df.to_csv(filename, index=False)
        print(f"Found {len(all_properties)} properties across {len(states)} states")
        return df

Scaling Across Counties

With over 3,000 counties in the US, scaling requires solid proxy infrastructure. ScraperAPI handles JavaScript rendering for modern county sites. ThorData residential proxies prevent IP blocks during large scans. ScrapeOps tracks success rates per county.

Legal Considerations

All data scraped here is public record. However, respect rate limits, comply with each site’s terms of service, and avoid overloading government infrastructure. This tool is designed for legitimate research, journalism, and investment analysis.

Track foreclosures systematically and you’ll see market shifts before they appear in the headlines.

How to Scrape Conference Speaker Lineups for Trend Detection

Conference speaker lineups are a leading indicator of industry trends. When multiple conferences simultaneously feature talks on a topic, it signals emerging demand months before mainstream adoption. Let’s build a scraper that tracks speaker lineups across tech conferences and identifies trending topics.

Why Conference Data Matters

By the time a topic appears in a Gartner report, it’s already mainstream. Conference organizers curate lineups 3-6 months ahead based on what’s gaining traction. Tracking these patterns gives you an early warning system for industry trends.

Building the Conference Scraper

import requests
from bs4 import BeautifulSoup
import re
from urllib.parse import urljoin

SCRAPER_API_KEY = "YOUR_KEY"

class ConferenceScraper:
    def scrape_speakers(self, conference_url):
        response = requests.get(
            "http://api.scraperapi.com",
            params={
                "api_key": SCRAPER_API_KEY,
                "url": conference_url,
                "render": "true"
            },
            timeout=60
        )
        soup = BeautifulSoup(response.text, "html.parser")
        speaker_links = []
        for link in soup.find_all("a", href=True):
            href = link["href"].lower()
            text = link.get_text().lower()
            if any(kw in href or kw in text for kw in ["speaker", "schedule", "agenda", "session"]):
                full_url = urljoin(conference_url, link["href"])
                speaker_links.append(full_url)

        speakers = []
        for page_url in set(speaker_links[:5]):
            page_speakers = self._scrape_speaker_page(page_url)
            speakers.extend(page_speakers)
        return speakers

    def _scrape_speaker_page(self, url):
        response = requests.get(
            "http://api.scraperapi.com",
            params={"api_key": SCRAPER_API_KEY, "url": url, "render": "true"},
            timeout=60
        )
        soup = BeautifulSoup(response.text, "html.parser")
        speakers = []
        cards = soup.select(
            ".speaker-card, .speaker-item, .speaker, "
            "[class*='speaker'], [class*='presenter']"
        )
        for card in cards:
            name_el = card.select_one("h2, h3, h4, .speaker-name, .name")
            title_el = card.select_one(".title, .role, .position, .company")
            talk_el = card.select_one(".talk-title, .session-title, .topic")
            if name_el:
                speakers.append({
                    "name": name_el.get_text(strip=True),
                    "title": title_el.get_text(strip=True) if title_el else "",
                    "talk": talk_el.get_text(strip=True) if talk_el else "",
                    "source_url": url
                })
        return speakers

Topic Extraction and Clustering

from sklearn.feature_extraction.text import TfidfVectorizer
from collections import Counter
import numpy as np

class TopicAnalyzer:
    STOP_WORDS = {
        "talk", "session", "keynote", "workshop", "panel",
        "introduction", "building", "using", "deep", "dive"
    }

    def extract_topics(self, speakers):
        talks = [s["talk"] for s in speakers if s.get("talk")]
        if not talks:
            return []
        vectorizer = TfidfVectorizer(
            max_features=1000, stop_words="english",
            ngram_range=(1, 3), min_df=2
        )
        tfidf = vectorizer.fit_transform(talks)
        feature_names = vectorizer.get_feature_names_out()
        mean_scores = np.asarray(tfidf.mean(axis=0)).flatten()
        top_indices = mean_scores.argsort()[-30:][::-1]
        topics = []
        for idx in top_indices:
            term = feature_names[idx]
            if term.lower() not in self.STOP_WORDS:
                topics.append({
                    "term": term,
                    "score": float(mean_scores[idx]),
                    "frequency": int((tfidf[:, idx] > 0).sum())
                })
        return topics

    def detect_trends(self, current_topics, previous_topics):
        current_dict = {t["term"]: t["score"] for t in current_topics}
        previous_dict = {t["term"]: t["score"] for t in previous_topics}
        trends = []
        for term, score in current_dict.items():
            prev_score = previous_dict.get(term, 0)
            if prev_score == 0 and score > 0.01:
                trends.append({"term": term, "type": "NEW", "score": score})
            elif score > prev_score * 1.5:
                growth = (score - prev_score) / max(prev_score, 0.001)
                trends.append({"term": term, "type": "GROWING", "growth": round(growth, 2)})
        return sorted(trends, key=lambda x: x.get("score", x.get("growth", 0)), reverse=True)

Cross-Conference Analysis

def cross_conference_analysis(conference_data):
    topic_conference_map = {}
    for conf_name, speakers in conference_data.items():
        analyzer = TopicAnalyzer()
        topics = analyzer.extract_topics(speakers)
        for topic in topics:
            term = topic["term"]
            if term not in topic_conference_map:
                topic_conference_map[term] = []
            topic_conference_map[term].append({
                "conference": conf_name,
                "score": topic["score"]
            })
    trending = {
        term: confs for term, confs in topic_conference_map.items()
        if len(confs) >= 3
    }
    return dict(sorted(trending.items(), key=lambda x: len(x[1]), reverse=True))

Scaling with Proxy Infrastructure

Scraping dozens of conference sites requires reliable proxies. ScraperAPI handles JavaScript-rendered conference pages. For geo-specific conferences, ThorData residential proxies work well. ScrapeOps monitors scraping health.

Running the Full Pipeline

conferences = {
    "PyCon 2026": "https://us.pycon.org/2026/schedule/",
    "KubeCon EU": "https://events.linuxfoundation.org/kubecon/",
    "AWS re:Invent": "https://reinvent.awsevents.com/sessions/"
}

scraper = ConferenceScraper()
all_data = {}
for name, url in conferences.items():
    speakers = scraper.scrape_speakers(url)
    all_data[name] = speakers
    print(f"{name}: {len(speakers)} speakers")

trends = cross_conference_analysis(all_data)
for topic, confs in list(trends.items())[:10]:
    print(f"TRENDING: {topic} ({len(confs)} conferences)")

Conference lineups are one of the most underutilized data sources in tech. Start tracking them systematically and you’ll spot trends months before they hit mainstream awareness.

Students miss out on billions in unclaimed scholarships every year simply because they don’t know they exist. Let’s build a scholarship matcher that scrapes award databases and matches students to opportunities they qualify for.

The Scholarship Discovery Problem

There are over 1.7 million private scholarships worth $7.4 billion annually. Most students only apply to a handful because finding relevant ones is tedious. A scholarship matcher solves this by aggregating data from hundreds of sources and using NLP to match student profiles.

Data Sources

Key scholarship databases to scrape:

• Fastweb, Scholarships.com, Cappex
• College Board Scholarship Search
• State-specific scholarship portals
• Professional association awards
• Community foundation directories

Setting Up

pip install requests beautifulsoup4 pandas scikit-learn schedule

Using ScraperAPI for JavaScript-heavy scholarship sites:

import requests
from bs4 import BeautifulSoup
import json

SCRAPER_KEY = "YOUR_SCRAPERAPI_KEY"

def fetch(url):
    resp = requests.get(
        "http://api.scraperapi.com",
        params={"api_key": SCRAPER_KEY, "url": url, "render": "true"},
        timeout=60
    )
    return BeautifulSoup(resp.text, "html.parser")

Scraping Scholarship Listings

def scrape_scholarships_com(category, page=1):
    """Scrape scholarships from Scholarships.com."""
    url = f"https://www.scholarships.com/financial-aid/college-scholarships/scholarships-by-type/{category}/page/{page}"
    soup = fetch(url)

    scholarships = []
    for item in soup.select(".scholarship-item, .award-listing"):
        name = item.select_one(".scholarship-name, h3")
        amount = item.select_one(".award-amount, .amount")
        deadline = item.select_one(".deadline")
        desc = item.select_one(".description, .snippet")
        link = item.select_one("a")

        if name:
            scholarships.append({
                "name": name.text.strip(),
                "amount": amount.text.strip() if amount else "Varies",
                "deadline": deadline.text.strip() if deadline else "Varies",
                "description": desc.text.strip() if desc else "",
                "url": link["href"] if link else "",
                "source": "scholarships.com"
            })
    return scholarships

# Scrape multiple categories
categories = [
    "merit-scholarships",
    "need-based-scholarships",
    "minority-scholarships",
    "stem-scholarships",
    "creative-arts-scholarships"
]

all_scholarships = []
for cat in categories:
    results = scrape_scholarships_com(cat)
    all_scholarships.extend(results)
    print(f"{cat}: {len(results)} scholarships")

Extracting Eligibility Criteria

import re

def extract_eligibility(description):
    """Parse eligibility criteria from description text."""
    criteria = {
        "min_gpa": None,
        "max_age": None,
        "citizenship": [],
        "majors": [],
        "states": [],
        "demographics": []
    }

    # GPA requirements
    gpa_match = re.search(r"(d.d+)s*GPA", description)
    if gpa_match:
        criteria["min_gpa"] = float(gpa_match.group(1))

    # Citizenship
    if "US citizen" in description.lower():
        criteria["citizenship"].append("US")
    if "permanent resident" in description.lower():
        criteria["citizenship"].append("PR")

    # Major keywords
    major_keywords = [
        "engineering", "computer science", "biology",
        "business", "nursing", "education", "mathematics"
    ]
    for major in major_keywords:
        if major in description.lower():
            criteria["majors"].append(major)

    return criteria

Building the Student Profile Matcher

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd

class ScholarshipMatcher:
    def __init__(self, scholarships_df):
        self.df = scholarships_df
        self.vectorizer = TfidfVectorizer(
            stop_words="english",
            max_features=5000
        )
        self.tfidf_matrix = self.vectorizer.fit_transform(
            self.df["description"]
        )

    def match(self, student_profile, top_n=20):
        """Find best matching scholarships for a student."""
        profile_text = self._profile_to_text(student_profile)
        profile_vec = self.vectorizer.transform([profile_text])

        scores = cosine_similarity(
            profile_vec, self.tfidf_matrix
        ).flatten()

        self.df["match_score"] = scores
        results = self.df.sort_values("match_score", ascending=False)

        # Filter by hard eligibility criteria
        results = self._apply_filters(results, student_profile)
        return results.head(top_n)

    def _profile_to_text(self, profile):
        """Convert student profile to searchable text."""
        parts = [
            profile.get("major", ""),
            profile.get("interests", ""),
            profile.get("background", ""),
            profile.get("career_goals", ""),
            " ".join(profile.get("activities", []))
        ]
        return " ".join(parts)

    def _apply_filters(self, df, profile):
        """Filter out scholarships student doesn't qualify for."""
        student_gpa = profile.get("gpa", 4.0)
        filtered = df[
            df["criteria"].apply(
                lambda c: c.get("min_gpa") is None or
                c["min_gpa"] <= student_gpa
            )
        ]
        return filtered

Usage Example

# Student profile
student = {
    "name": "Alex Chen",
    "gpa": 3.8,
    "major": "computer science",
    "interests": "artificial intelligence machine learning",
    "background": "first-generation college student",
    "career_goals": "AI research in healthcare",
    "activities": [
        "robotics club president",
        "hospital volunteer",
        "coding bootcamp mentor"
    ],
    "state": "CA",
    "citizenship": "US"
}

# Load scholarships
df = pd.read_csv("scholarships.csv")
df["criteria"] = df["description"].apply(extract_eligibility)

matcher = ScholarshipMatcher(df)
matches = matcher.match(student, top_n=15)

print("Top Scholarship Matches:")
for _, row in matches.iterrows():
    print(f"  {row['name']} - {row['amount']} ({row['match_score']:.0%} match)")

Keeping Data Fresh

Scholarship deadlines change every year. Set up weekly scraping to keep your database current. Use ThorData for rotating proxies across scholarship sites, and ScrapeOps to monitor which scrapers need maintenance.

import schedule

def weekly_refresh():
    for source in SCHOLARSHIP_SOURCES:
        try:
            new_data = scrape_source(source)
            update_database(new_data)
            print(f"Updated {source}: {len(new_data)} scholarships")
        except Exception as e:
            print(f"Failed {source}: {e}")

schedule.every().monday.at("03:00").do(weekly_refresh)

Conclusion

A scholarship matcher can help students discover thousands of dollars in funding they’d otherwise miss. The combination of web scraping for data collection and NLP for matching creates a powerful tool. Start with 2-3 major databases, build the matching engine, then expand your sources over time.

The $7.4 billion in annual scholarships is there for the taking. The hard part isn’t qualifying. It’s knowing they exist.

A year ago, if you wanted AI help writing code, you opened your editor and used whatever was built in. Copilot, Cursor, Windsurf. They’ve gotten genuinely capable. Agent modes that make multi-file changes, run commands, iterate on errors. These aren’t just autocomplete anymore.

But something else was gaining traction at the same time, and it’s quietly changing how serious AI-assisted development actually works.

The CLI agents

Claude Code, OpenAI’s Codex CLI, OpenCode, and others work directly in your terminal. They operate in your actual development environment with full access to the filesystem, your shell, your toolchain. They read your codebase, create files, run tests, fix what breaks, and iterate until the job is done.

Editor agent modes can do a lot of this too. The difference isn’t really about what the agent can do in theory. It’s about what model is doing the work and what it costs you to use it.

The model economics

Every tool in this space is grappling with the same problem: the best AI models are expensive to run, and someone has to pay for them.

Editor-embedded tools bundle model access into their subscription. Cursor uses a credit system (introduced June 2025) where credits deplete at different rates depending on which model handles the request. Their Pro plan is $20/month, Ultra is $200/month with a much larger credit pool. If you exceed your credits, you pay overages at API rates. Their agent and edit features only work with Cursor’s own custom models. You can’t bring your own API key for those.

Windsurf recently restructured to a quota system with daily and weekly refreshes. Pro is $20/month, and they added a Max tier at $200/month for heavier usage. Individual users can still bring their own API keys, but teams and enterprise users can’t.

GitHub Copilot uses premium request allowances. Copilot Pro+ at $39/month includes 1,500 premium requests, with overages at $0.04 each. When you exceed your allowance without paying overages, you fall back to a less capable model.

CLI agents connect to model providers directly. Claude Code authenticates with a Claude Max subscription ($100 or $200/month), giving you Opus with weekly limits. Codex CLI can authenticate with your ChatGPT Pro subscription ($200/month) for GPT-5.4 Pro access. You can also use API keys with per-token billing if you prefer.

At the $200/month price point, both approaches have limits. Claude Max and ChatGPT Pro use rolling time windows (5-hour and daily resets). Cursor Ultra and Windsurf Max use credit or quota pools. Both sides offer the option to pay overages at API rates when you exceed your allowance. The details of how much usage you actually get for $200/month are hard to compare directly since each platform measures differently, but anecdotally the model provider subscriptions tend to be more generous for heavy agent use than the equivalent editor tier.

The other difference is model choice. With CLI agents, you pick your model and your provider. Claude Code runs Opus because that’s what Anthropic offers through Claude Max. Codex CLI runs whatever OpenAI makes available through ChatGPT Pro. With editor subscriptions, you use whatever models the editor vendor has chosen to offer through their system, and for Cursor specifically, agent features only work with their custom models.

Context

Every AI coding tool manages context, and every one of them hits limits eventually. CLI agents are no exception. Claude Code compacts conversation history when sessions get long. Auto-compaction kicks in around 80% of the context window, and earlier parts of the conversation get summarized. This is a real limitation that affects output quality over long sessions.

Editor-embedded tools face the same fundamental constraint, with an additional layer of complexity. They’re assembling context from codebase indexes, open files, retrieval systems, and file references. Some show a usage meter so you can see the context window filling up. The context management is sophisticated, but it’s also something you end up thinking about. Which files to reference, when to start a fresh session, how to keep the AI aware of what matters.

CLI agents have a more direct relationship with your project. The agent reads files from the filesystem when it needs them, rather than depending on what a retrieval system surfaced or which files happen to be open. The context window sizes are comparable when using the same underlying models, but you tend to spend less time managing the AI’s awareness of your project and more time on the actual problem.

The workflow gap

Here’s the thing about CLI agents. They’re terminals. You start a session, the agent works, and then you’re left with a pile of changes in a directory somewhere. Turning that into a reviewed, tested, merged pull request is still on you. And if you want to run multiple agents in parallel on the same repo? Good luck managing the git conflicts.

This is the gap that desktop AI workspaces fill. Not by replacing the CLI agents, but by giving them a proper environment to work in.

At Taskeract, we built the layer that wraps around these agents. Every session gets its own isolated git worktree, so agents never step on each other’s work, or yours. You can run Claude Code in one session and Codex in another, both working on the same project simultaneously, on separate branches that won’t conflict.

But isolation is just the starting point. The real workflow starts when the code is written.

From issue to done

Most tools in this space handle some of the post-coding workflow. You can create PRs, review diffs, push changes. But the full loop (starting from an issue, reviewing changes, creating a PR, monitoring CI, responding to reviewer feedback, merging, and closing the issue) still involves jumping between multiple tools. Your terminal, your browser, your git hosting provider, your issue tracker.

We built Taskeract to cover the entire loop. Start a session from an issue in GitHub, GitLab, Jira, Linear, or Trello. The agent works in its isolated environment. When it’s done, review the changes with syntax-highlighted diffs right in the app. Push, create a PR, see CI status, respond to review threads, and merge, all without leaving the window. The issue automatically advances through its workflow states as the work progresses.

It’s the difference between a tool that helps you write code and a tool that helps you ship code.

Where this is going

The autonomous agent approach with top-tier models is already producing better results at more predictable costs. And once the code is written, the workflow around it matters just as much as the code itself.

If you’ve been feeling like AI coding hasn’t quite lived up to the promise, it might not be the AI that’s the bottleneck. It might be what’s around it.

The shift already happened. The question is whether your workflow has caught up.

CSI Drivers (How Storage Actually Works in EKS)

So far:

• VPC → network
• IAM → permissions
• EKS → control plane
• Nodes → compute

Now comes the part many people ignore:

👉 Storage

Without this:

• Pods can’t persist data
• Databases won’t work
• Logs disappear on restart

This module installs CSI drivers, which allow Kubernetes to use AWS storage.

📂 Module Files

modules/csi-driver/
├── main.tf
├── variables.tf
└── outputs.tf

📄 variables.tf

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "ebs_csi_role_arn" {
  description = "IAM role ARN for EBS CSI driver"
  type        = string
}

variable "s3_csi_role_arn" {
  description = "IAM role ARN for S3 CSI driver"
  type        = string
}

variable "node_group_dependency" {
  description = "Node group dependency to ensure nodes exist first"
  type        = any
}

🧠 What these inputs mean

• cluster_name → where to install addons
• ebs_csi_role_arn → IAM role for EBS driver
• s3_csi_role_arn → IAM role for S3 driver
• node_group_dependency → ensures nodes exist first

⚠️ Important insight

This module depends on:

👉 IAM (for IRSA roles)
👉 Node groups (for scheduling pods)

📄 main.tf

1. EBS CSI Driver

resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = var.cluster_name
  addon_name               = "aws-ebs-csi-driver"
  service_account_role_arn = var.ebs_csi_role_arn

  depends_on = [var.node_group_dependency]
}

🧠 What this does

Installs:

👉 AWS EBS CSI Driver inside cluster

What is CSI?

CSI = Container Storage Interface

👉 It allows Kubernetes to talk to AWS storage.

What EBS CSI enables

• Create EBS volumes
• Attach volumes to pods
• Persist data

Important line

service_account_role_arn = var.ebs_csi_role_arn

👉 This is IRSA

This means:

• Pod gets IAM role
• Pod can call AWS APIs

Without this

❌ Pod cannot create volumes
❌ PVC fails

2. S3 CSI Driver

resource "aws_eks_addon" "s3_csi" {
  cluster_name             = var.cluster_name
  addon_name               = "aws-mountpoint-s3-csi-driver"
  service_account_role_arn = var.s3_csi_role_arn

  depends_on = [var.node_group_dependency]
}

🧠 What this does

Installs:

👉 S3 CSI driver

What it enables

Mount S3 bucket as:

Pod → S3 bucket (like filesystem)

Use cases

• logs
• shared storage
• backups

3. Dependency Handling

depends_on = [var.node_group_dependency]

Why this is critical

CSI driver runs as pods.

👉 Pods need nodes

So order must be:

Nodes → CSI Driver

Without this

• Addon installs
• But pods fail to schedule

📄 outputs.tf

output "ebs_csi_addon_id" {
  description = "EBS CSI addon ID"
  value       = aws_eks_addon.ebs_csi.id
}

output "s3_csi_addon_id" {
  description = "S3 CSI addon ID"
  value       = aws_eks_addon.s3_csi.id
}

🧠 Why outputs matter

Used for:

• tracking addon deployment
• debugging
• dependencies in future modules

🔥 What You Actually Built

Kubernetes Pod
      │
      │
CSI Driver
      │
      │
AWS Storage (EBS / S3)

⚠️ Real Issues People Face

• Missing IRSA → access denied
• No node dependency → pods fail
• Wrong role → volume attach fails
• Forgetting CSI → PVC stuck in pending

🧠 Key Takeaways

• Kubernetes doesn’t manage storage directly
• CSI drivers connect Kubernetes to AWS
• IRSA is required for secure access
• EBS = block storage
• S3 = object storage

🚀 Next

In Part 6:

👉 AWS Load Balancer Controller
👉 How ALB actually integrates with Kubernetes
👉 Why target-type: ip matters

At this point, your cluster can:

• run workloads
• persist data

Now we move to traffic layer.