Описание
The project builds a solution that automatically discovers, extracts, normalizes, and structures technical product and asset data from publicly available internet sources. It creates structured catalogs of manufacturers, product lines, models, and technical specifications, delivers standardized CSV files, and maps the data to an existing EAM structure.
Задачи
•Identify and evaluate relevant public data sources, assessing their coverage, quality, freshness, and technical accessibility
•Analyze HTML, DOM, tables, PDFs, download structures, APIs, and manufacturer, product, and model coverage
•Consider technical and legal constraints related to data collection
•Design scalable web crawling and scraping architecture
•Build API and file-based ingestion flows
•Extract data from web pages, tables, PDFs, and technical documents
•Define AI-assisted extraction and normalization workflows
•Implement validation, duplicate detection, source traceability, and quality controls
•Build modular, extensible crawlers, parsers, and connectors
•Develop components for source discovery, extraction, mapping, and normalization
•Support heterogeneous website structures and document formats
•Extract manufacturers, product families, types, and technical attributes using LLM/NLP methods
•Transform extracted content into structured domain objects and map attributes to the existing EAM structure
•Build standardized CSV outputs and maintain metadata and source traceability
•Process large datasets using Python, Pandas, PySpark, and Databricks
•Implement completeness and plausibility checks
•Build a Databricks application for domain validation
•Flag uncertain or low-confidence extracted values
•Support correction, approval, and error classification workflows, enabling technical validation by subject-matter experts
•Run pilots for selected asset classes, manufacturers, and sources
•Measure extraction accuracy and validation quality, identify optimization opportunities, and define scaling requirements
•Document crawlers, connectors, extraction logic, architecture, deployment, the Databricks application, data model, and CSV interface
•Deliver source code and deployment configuration
Требования
•Strong Python development experience
•Experience with Pandas, PySpark, Databricks, Delta Lake, and Databricks Apps
•Strong experience with Scrapy, Beautiful Soup, or comparable frameworks
•Experience analyzing HTML, DOM, and tables; handling dynamic websites and download structures; and building robust, extensible crawlers and parsers
•Hands-on experience with LLMs and NLP, prompt engineering, structured output generation, technical information extraction and normalization, classification and mapping, duplicate detection, and entity resolution
•Experience with REST APIs, file-based data ingestion, PDF and datasheet extraction, table extraction, schema mapping, source and metadata management, and extraction pipeline error handling and monitoring
•Fluent English
Будет плюсом
•MLOps and CI/CD for Data/AI solutions
•Selenium/Playwright or similar browser automation tools
•Azure services
•Databricks deployment experience
•Product, asset, or technical type catalogs, ontologies and semantic models, technical feature catalogs
•EAM/SAP attribute models
•Electrical engineering domain knowledge
•Experience designing crawling solutions in line with data protection and legal requirements
Условия
•Long-term international project
•25 calendar days of vacation and 5 additional paid sick days
•Medical insurance
•Corporate English courses
•Corporate events and team-building activities
•Support with professional certifications
•Reimbursement for professional courses and training
•Opportunities for professional and technical growth
