Before the LLM boom, AI meant something different to me. Understanding intelligence. Building minds that think. Deep questions about consciousness, reasoning, learning.
Then ChatGPT happened. Suddenly everyone’s an “AI engineer” for calling APIs. Billions of dollars. Closed models. Corporate control. The questions I cared about disappeared from conversation.
Now studying AI feels like surrendering to hype instead of pursuing genuine curiosity.
Only large companies control the cutting edge. OpenAI, Google, Meta. Billions in compute. Closed research. It feels like only money matters.
The most interesting work isn’t at these companies though. It’s in open source projects, individual researchers, problems the big companies ignore.
Corporate dominance is deployment, not innovation. Real advancement comes from understanding fundamentals, novel architectures, research without immediate commercial value.
Should I study LLMs to catch up? Wrong frame.
The question is: do I want to understand how intelligence works?
LLMs might reveal something about intelligence. Or they might just be very good pattern matchers that don’t actually think. I don’t know yet.
That’s the tension. The questions I care about are about real intelligence. Whether current AI gets me closer to those answers - I’m still figuring that out.
If I study this, it’s through implementation. Build a transformer from scratch. Implement attention mechanisms. Train small models. Experiment with novel architectures.
The genuine AI research is still happening. Consciousness in machines. How cognition emerges from computation. Novel architectures that think differently. Understanding intelligence from first principles.
I want to reconnect with the AI that matters. The questions that made me interested before the hype, before the corporate takeover.
How does a mind work? Can consciousness emerge from computation? What is intelligence? How do you build something that truly thinks?
LLMs might be relevant to these questions. Or they might not. But the questions themselves are eternal.
My real interest didn’t disappear. It got buried under frustration. The challenge is finding the path back to genuine curiosity.
The AI I care about still exists. It’s just not trending on Twitter/X.
that was the end of the blog;
Shortly after publishing, I found this. It answered most of my questions, it might answer yours too!
Playwright is 30% faster than Selenium with 50% fewer timeout errors. Firefox has an 85% detection rate with Chrome versus 15% when you switch. UV installs packages in 3 seconds instead of 67.
The numbers matter because you’ll be running these systems constantly. Small inefficiencies compound.
async with async_playwright() as p:
browser = await p.firefox.launch(headless=True)
page = await browser.new_page()
await page.click('button:has-text("Load More")')
Playwright auto-waits. This eliminates 90% of timing bugs. Selenium makes you handle this manually with explicit waits and arbitrary sleep() calls.
Firefox works because anti-bot systems optimize for Chrome (70% market share). They spend less effort detecting Firefox. The Gecko engine also produces different fingerprints than Blink.
UV is just better package management. Run uv add playwright and you’re done. Pip takes forever and creates non-deterministic environments.
Modern sites fingerprint everything. Canvas rendering, WebGL, audio context, font enumeration. Reusing the same context creates patterns.
# One browser, multiple contexts
browser = await p.firefox.launch()
context1 = await browser.new_context()
page1 = await context1.new_page()
context2 = await browser.new_context()
page2 = await context2.new_page()
Each context is a clean slate. No cookies, no localStorage, unique fingerprint. You can run them in parallel. Memory doesn’t leak across contexts.
Most modern sites load data via API. Scraping HTML is slow and fragile. The API is faster and returns structured JSON.
Open the network tab. Look for XHR/Fetch requests. Common patterns: /api/, /graphql/, api.bazaarvoice.com. Capture the request headers and parameters. Replay them directly.
class APIDiscovery:
def setup_monitoring(self, page):
page.on('request', self.capture_request)
async def capture_request(self, request):
url = request.url
if 'api' in url.lower():
print(f"Found: {request.method} {url}")
print(f"Headers: {dict(request.headers)}")
Many sites use third-party review systems (Bazaarvoice, Trustpilot). These have public APIs. Hit them directly instead of parsing their JavaScript widgets.
When you find the API, you’re 10x faster than scraping HTML. No CSS selectors to maintain. No waiting for JavaScript to render.
Clicking pagination buttons is slow and breaks easily. Understand the URL pattern instead.
# Instead of clicking 50 times
patterns = [
f"{base_url}?page={i}",
f"{base_url}/p/{i}",
]
# Direct navigation
for page_num in range(1, 51):
await page.goto(f"{base_url}?page={page_num}")
You can now parallelize. Process page 1, 10, and 20 simultaneously. No sequential clicking. If you know there are 50 pages, you’re done in minutes instead of hours.
Perfect timing and instant form filling looks robotic. Add variation.
async def human_type(page, selector, text):
await page.click(selector)
await page.wait_for_timeout(random.randint(100, 300))
for char in text:
await page.type(selector, char, delay=random.randint(50, 150))
if random.random() < 0.05: # 5% typo rate
await page.type(selector, random.choice('abcdefgh'))
await page.wait_for_timeout(random.randint(200, 500))
await page.press(selector, 'Backspace')
await page.wait_for_timeout(random.randint(500, 1500))
Variable typing speed. Occasional typos. Natural pauses. This matters more than you think.
Production scraping fails constantly. Timeouts, blocks, layout changes, network issues.
async def scrape_with_retry(url, max_retries=3):
for attempt in range(max_retries):
try:
async with async_playwright() as p:
browser = await p.firefox.launch()
context = await browser.new_context()
page = await context.new_page()
await page.goto(url, wait_until='networkidle')
data = await extract_data(page)
return data
except TimeoutError:
wait_time = 2 ** attempt
await asyncio.sleep(wait_time)
except Exception as e:
if "blocked" in str(e).lower():
context = await browser.new_context() # Fresh context
else:
raise
Different errors need different strategies. Timeouts get exponential backoff. Blocks get fresh contexts. Network errors retry immediately. Missing elements skip gracefully.
When something breaks in production, you need to see what happened.
async def debug_scrape(page, milestone):
timestamp = datetime.now().strftime("%H%M%S")
await page.screenshot(path=f"debug/{milestone}_{timestamp}.png")
Capture screenshots before/after key interactions. On errors. When selectors fail. These saved me hours of debugging blind.
Hard-coding selectors makes your scraper unmaintainable.
{
"amazon_us": {
"selectors": {
"search_box": "input#twotabsearchtextbox",
"product_links": "a[data-component-type='s-search-result']",
"price": ".a-price-whole"
},
"delays": {
"min": 1000,
"max": 3000
}
}
}
When Amazon changes their layout, update the JSON. No code deployment. Non-developers can maintain selectors. You can A/B test different strategies by swapping configs.
Too many concurrent requests get you blocked. Too few waste time.
class AdaptiveConcurrency:
def __init__(self, initial_limit=5):
self.limit = initial_limit
self.error_rate = 0
def adjust(self):
if self.error_rate > 0.10:
self.limit = max(1, self.limit - 1)
elif self.error_rate < 0.02:
self.limit += 1
Start with 5 concurrent requests. If errors exceed 10%, reduce. If errors stay below 2%, increase. The system finds its own equilibrium.
Production scraping is about resilience, not extraction. Sites change constantly. Anti-bot measures evolve. Your scraper needs to adapt.
The goal isn’t extracting data once. It’s building systems that extract data reliably over years. Focus on observability, maintainability, and respect for target sites.
Use modern tools. Create fresh contexts. Find the APIs. Handle errors well. Take screenshots. Configure everything. Control concurrency.
These patterns work because they’ve survived production.
]]>def print_hi(name)
puts "Hi, #{name}"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.I have kept the original default post by Jekyll so that I can return and relive this moment of creating this !!
]]>