Senior Software Engineer (Web Scraping, Data acquisition, Data collection)
- Posted last month · via SmartRecruiters
Hiring in some regions only
Design scalable web scraping architectures across thousands of domains
Build and lead distributed scraping projects, detecting anti-bot measures and setting up anomaly detection. Work with large-scale scraping projects and handle proxies, session management, and TLS tweaking. Deliver scalable and reliable data collection solutions.
Why This Role?
Ownership and freedom to experiment, backed by a team that thrives on innovation
Key Responsibilities
- Design and build distributed scraping architectures
- Reverse engineer APIs and mobile apps
- Detect common anti-bot measures and build crawlers that adjust to appear more human-like
Requirements
- 5+ years of experience with Python and large-scale scraping projects
- Deep understanding of anti-bot technologies and how to bypass them
- Experience working with Scrapy, Requests, httpx, Selenium, Playwright, or custom headless browser drivers
Required Skills
Indonesia Context
- Working Hours Overlap:
- Flexible — work your own hours
View Original Description from SmartRecruitersShow more
Original description from SmartRecruiters
Design and build distributed scraping architectures that can scale across thousands of domains. Reverse engineer APIs and mobile apps using tools like Frida , mitmproxy , Charles Proxy, burp and many more. Detect common anti-bot measures (like JavaScript checks or browser behavior detection) to build crawlers that adjust to appear more human-like. Lead technical innovation in scraping: identify edge cases, prototype new approaches, and share knowledge with the team. Set up anomaly detection and monitoring to detect blockers or failures early 5+ years of experience with Python and large-scale scraping projects. Deep understanding of anti-bot technologies and how to bypass them, fingerprinting, WAFs, JavaScript challenges, etc. Experience working with Scrapy , Requests , httpx , Selenium , Playwright , or custom headless browser drivers. Skilled at handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking. Strong debugging skills across browser, HTTP traffic, and device-level interactions. You think independently and enjoy solving problems no one has solved before. Familiarity with structured storage formats (Parquet, Avro), or real-time stream processing. Good knowledge of databases like MongoDB, InfluxDB, and Redis, including how they work and how to optimize them. ➕ Bonus Skills Experience with real-device simulation for mobile scraping. Comfortable building and maintaining distributed task queues (Celery, Kafka, etc.). Experience scraping high-security platforms or mobile-first environments. Exposure to Linux or cloud environments (AWS, GCP, etc.). Basic understanding of Kubernetes, Cloud Run, and Cloud Functions to deploy and orchestrate scraping workloads. Why Join NIQ Digital shelf You'll have ownership and freedom to experiment, backed by a team that thrives on innovation. You'll work with people who are truly passionate about data and know how to use it well. Your ideas will shape our strategy, tools, and success — you won’t just follow the Playbook; you’ll write about it. Friendly and transparent work culture, competitive salary, flexible hours, and the chance to grow fast in a high-impact role. Flexible working environment Volunteer time off LinkedIn Learning Employee-Assistance-Program (EAP) Our Benefits Flexible working environment Volunteer time off LinkedIn Learning Employee-Assistance-Program (EAP) NIQ may utilize artificial intelligence (AI) tools at various stages of the recruitment process, including résumé screening, candidate assessments, interview scheduling, job matching, communication support, and certain administrative tasks that help streamline workflows. These tools are intended to improve efficiency and support fair and consistent evaluation based on job-related criteria. All use of AI is governed by NIQ’s principles of fairness, transparency, human oversight, and inclusion. Final hiring decisions are made exclusively by humans. NIQ regularly reviews its AI tools to help mitigate bias and ensure compliance with applicable laws and regulations. If you have questions, require accommodations, or wish to request human review were permitted by law, please contact your local HR representative. For more information, please visit NIQ’s AI Safety Policies and Guiding Principles: https://nielseniq.com/global/en/info/niqs-ai-safety-policies/ About NIQ NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population. For more information, visit NIQ.com Want to keep up with our latest updates? Follow us on: LinkedIn | Instagram | Twitter | Facebook Our commitment to Diversity, Equity, and Inclusion At NIQ, we are steadfast in our commitment to fostering an inclusive workplace that mirrors the rich diversity of the communities and markets we serve. We believe that embracing a wide range of perspectives drives innovation and excellence. All employment decisions at NIQ are made without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, marital status, veteran status, or any other characteristic protected by applicable laws. We invite individuals who share our dedication to inclusivity and equity to join us in making a meaningful impact. To learn more about our ongoing efforts in diversity and inclusion, please visit the https://nielseniq.com/global/en/news-center/diversity-inclusion
Not quite right? Similar roles hiring now
See all 5 similar →Employer hires only in specific regions — check eligibility first
Explore related
Market data & reports
Salary & skill-demand research built from our own listings data.
- Skills Remote Employers Ask For in the AI EraMost-named skills in active listings by kind, AI-named skills and the fastest risers. Weekly data.
- Indonesia IT Jobs vs Global Remote (2026)Primary analysis of 2,049 listings: methodology, classification rules, downloadable datasets.
- AI-Skill Demand: Indonesia vs Global Remote (2026)10,000+ postings, taxonomy-first classifier, Wilson CIs, pre-registered before analysis.
- Remote ≠ Remote: The Skills That Open Global Work to Indonesians (2026)12,891 remote listings: the highest-paid coding skills are the most geo-locked for Indonesia-based applicants. CC BY 4.0 aggregate dataset.
- The Compliance Layer of the AI Hiring Stack (2026)6,349 remote listings: 77.2% never state who may apply. Methodology and a CC BY 4.0 aggregate dataset.
- AI in Job Ads: Indonesia vs Global (2026)Global engineering ads naming AI rose from 3.4% to 23% after ChatGPT; Indonesian ads went from 2.2% to 2.8%. 2027 scenarios and a CC BY 4.0 dataset.
- Future of Jobs Index: every quarterly editionOne index per quarter from global remote and Indonesian job ads. Q4 2026 is the baseline (= 100); later editions are measured against it on the same sources.
- Future of Jobs Index, Q3 2026 (pre-baseline)Global remote and Indonesian job ads by job family: AI-skill demand, remote reach, pay transparency and seniority. Every figure with its sample size. CC BY 4.0 dataset.
- The Laptop Job Recession: Indonesia's Coders and Their Job Ads (2026)Indonesia's coders multiplied by eight while software job ads fell by two-thirds. 284,000 archived Southeast Asian job ads, 2019–2026, charts by job family and 2027 bets. CC BY 4.0 dataset.
- Pandora's Box Ep. 1: AI and Southeast Asian Job Ads (2026)118,545 archived job ads, 2019–2026: web and software developer ads shrank on every board, a phone app became the entry ticket, and a Job X-ray for 17 jobs. CC BY 4.0 datasets.
- Cron vs LLM vs LangGraph vs Agent for Social Posts (2026)A pre-registered test of four ways to turn a blog post into social posts, on two models. Automated checks picked a cron template; the model mattered more than the framework.
- Indonesia Hiring Report: Tech vs Non-TechJob demand by field from aggregate open-job counts — never individual listings.
- Indonesia Salary BenchmarkAggregate salary ranges across roles, with open methodology and dataset.
- Indonesian Remote Work Salary & Demand IndexHow much of the global remote job corpus is open to Indonesia, and what it pays (USD) by role.
- Indonesia Quarterly Labor Market ReportLayoffs, funding, salaries & skills per quarter — open aggregates.
- Remote Market Reports by RoleAuto-generated per role family — skills, seniority, companies, salary.
- Global Remote Salary BenchmarkAnnual salary by role & currency, plus the share of listings open worldwide.
This employer appears to hire only in certain regions.