Skip to content

Architecture Overview

This project implements a two-phase web scraping pipeline using AWS Lambda functions written in Node.js, Go and Perl. The system is designed to handle dynamic, JavaScript-rendered pages and convert them into structured data.

High-Level Flow

This diagram is generated from YAML using awsdac.

Web Scraper Architecture Diagram

Components

  1. fetcher-schedule

    • An EventBridge rule that triggers the fetcher-lambda on a defined schedule (e.g., daily)
    • Allows full decoupling of the fetching cadence from other processing components

  2. fetcher-lambda

    • A Node.js Lambda function responsible for fetching HTML content
    • Uses headless Chromium (via Puppeteer) to render JavaScript-heavy pages
    • Stores the fully rendered HTML in an intermediate S3 bucket

  3. web-scraper-output

    • Acts as a bridge between the fetch and parse phases
    • Receives raw HTML from the fetcher

  4. s3-parser-trigger

    • An EventBridge rule that listens for ObjectCreated events from the web-scraper-output S3 bucket
    • Allows for decoupled and scalable downstream processing

  5. parser-lambda

    • A Perl-based Lambda function and a Go Lambda function that parse the stored HTML
    • The Perl function uses a custom runtime implemented via a bootstrap script and is deployed as a Docker container, while the Go function uses a zip-based deployment with the provided.al2 runtime
    • Parses HTML using Mojo::DOM (Perl) or PuerkitoBio/goquery (Go) to extract structured job listings

  6. job-store

    • Stores structured job listings parsed by the Lambda functions

Infrastructure

  • terraform-web-scraper
    Terraform project for managing infrastructure: S3 buckets, Lambda deployment, IAM roles, and CI/CD wiring.