The Transformation layer converts raw, messy external data into our clean, internal ProductSchema.

The Transform Class

Defined in app/dataflow/etl_service.py, this class instantiates the correct Mapper based on the database configuration.

transformer = Transform(transform=data_transform_record)
result = await transformer.transform(raw_document, merchant_id=123)

Mappers

Mappers are located in app/dataflow/mappers/. They all follow a similar pattern: implement a transform(obj) -> ProductSchema method.

1. Shopify Mapper (shopify_product_map.py)

• Complexity: High.
• Variants: Handles the conversion of Shopify's "Options" (Option1=Red, Option2=Large) into our flat attributes list.
• Cleaning: Uses clean_html() to strip HTML tags from descriptions (Shopify stores HTML, we prefer text/markdown).
• Weights: specific logic to parse grams or weight + weight_unit into a unified string.
• Status: Maps "draft"/"active" to our internal ProductStatus enum.

2. Whop Mapper (whop_product_map.py)

• Concept: Whop sells "Access" (software), not physical goods.
• Plans as Variants: A Whop "Product" is the software; a "Plan" (Monthly, Lifetime) is the Variant (SKU).
• Attributes: Extracts SaaS-specific attributes:
  • plan_type (renewal vs one-time)
  • billing_period
  • trial_period_days
• Placeholder: If a product has no plans, it creates a "placeholder" inactive variant so the product still exists in our DB.

3. Petswyak Mapper (petswyak_product_map.py)

• Simplicity: Direct 1:1 mapping.
• Specifics: Handles free_qty for stock and concatenates weight + weight_uom.

4. AI Mapper (product_map.py)

• Role: The "Universal Adapter".
• Mechanism: Uses GPT-4o-mini (via ai_client) to intelligently parse any unknown JSON structure into our schema.
• Prompt: Uses SYSTEM_PRODUCT_MAP_PROMPT which instructs the AI to:
  • Infer currency from the merchant's country.
  • Extract attributes implicitly (e.g., "Blue Shirt" -> Color: Blue).
  • Map categorization to the merchant's existing categories.
• Use Case: Rapidly onboarding new merchants with non-standard APIs without writing code.