Senior Software Engineer (Web Scraping, Data acquisition, Data collection)
- Diposting bulan lalu · lewat SmartRecruiters
Rekrut di wilayah tertentu saja
Bangun arsitektur scraping yang dapat di-skalakan untuk ribuan domain
Sebagai Senior Software Engineer di NielsenIQ, kamu akan merancang dan membangun arsitektur scraping yang dapat di-skalakan untuk ribuan domain. Kamu akan membalikkan enjin API dan aplikasi mobile menggunakan berbagai alat. Kamu juga akan mendeteksi dan mengatasi anti-bot measures untuk membuat crawler yang lebih mirip manusia. Kamu akan bekerja dengan tim yang inovatif dan memiliki kebebasan untuk bereksperimen.
Kenapa Menarik?
Kamu akan memiliki kebebasan untuk bereksperimen dan memiliki dampak tinggi dalam peran ini
Tanggung Jawab Utama
- Membangun arsitektur scraping yang dapat di-skalakan untuk ribuan domain
- Membalikkan enjin API dan aplikasi mobile menggunakan alat seperti Frida, mitmproxy, dan Charles Proxy
- Mendeteksi dan mengatasi anti-bot measures untuk membuat crawler yang lebih mirip manusia
- Membangun dan memantau sistem deteksi anomali untuk mendeteksi blokir atau kegagalan awal
Persyaratan
- Pernah bekerja dengan Python dan proyek scraping besar selama 5+ tahun
- Memiliki pengalaman dengan Scrapy, Requests, httpx, Selenium, atau Playwright
- Memiliki kemampuan dalam mengelola proxy, manajemen sesi, dan cookie injection
- Memiliki pengetahuan tentang basis data seperti MongoDB, InfluxDB, dan Redis
Skills Wajib
Konteks Indonesia
- Overlap Jam Kerja:
- Fleksibel — atur jam kerjamu sendiri
Lihat Deskripsi Asli dari SmartRecruitersTampilkan selengkapnya
Deskripsi asli dari SmartRecruiters
Design and build distributed scraping architectures that can scale across thousands of domains. Reverse engineer APIs and mobile apps using tools like Frida , mitmproxy , Charles Proxy, burp and many more. Detect common anti-bot measures (like JavaScript checks or browser behavior detection) to build crawlers that adjust to appear more human-like. Lead technical innovation in scraping: identify edge cases, prototype new approaches, and share knowledge with the team. Set up anomaly detection and monitoring to detect blockers or failures early 5+ years of experience with Python and large-scale scraping projects. Deep understanding of anti-bot technologies and how to bypass them, fingerprinting, WAFs, JavaScript challenges, etc. Experience working with Scrapy , Requests , httpx , Selenium , Playwright , or custom headless browser drivers. Skilled at handling proxies (residential, datacenter, rotating), session management, cookie injection, and TLS tweaking. Strong debugging skills across browser, HTTP traffic, and device-level interactions. You think independently and enjoy solving problems no one has solved before. Familiarity with structured storage formats (Parquet, Avro), or real-time stream processing. Good knowledge of databases like MongoDB, InfluxDB, and Redis, including how they work and how to optimize them. ➕ Bonus Skills Experience with real-device simulation for mobile scraping. Comfortable building and maintaining distributed task queues (Celery, Kafka, etc.). Experience scraping high-security platforms or mobile-first environments. Exposure to Linux or cloud environments (AWS, GCP, etc.). Basic understanding of Kubernetes, Cloud Run, and Cloud Functions to deploy and orchestrate scraping workloads. Why Join NIQ Digital shelf You'll have ownership and freedom to experiment, backed by a team that thrives on innovation. You'll work with people who are truly passionate about data and know how to use it well. Your ideas will shape our strategy, tools, and success — you won’t just follow the Playbook; you’ll write about it. Friendly and transparent work culture, competitive salary, flexible hours, and the chance to grow fast in a high-impact role. Flexible working environment Volunteer time off LinkedIn Learning Employee-Assistance-Program (EAP) Our Benefits Flexible working environment Volunteer time off LinkedIn Learning Employee-Assistance-Program (EAP) NIQ may utilize artificial intelligence (AI) tools at various stages of the recruitment process, including résumé screening, candidate assessments, interview scheduling, job matching, communication support, and certain administrative tasks that help streamline workflows. These tools are intended to improve efficiency and support fair and consistent evaluation based on job-related criteria. All use of AI is governed by NIQ’s principles of fairness, transparency, human oversight, and inclusion. Final hiring decisions are made exclusively by humans. NIQ regularly reviews its AI tools to help mitigate bias and ensure compliance with applicable laws and regulations. If you have questions, require accommodations, or wish to request human review were permitted by law, please contact your local HR representative. For more information, please visit NIQ’s AI Safety Policies and Guiding Principles: https://nielseniq.com/global/en/info/niqs-ai-safety-policies/ About NIQ NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population. For more information, visit NIQ.com Want to keep up with our latest updates? Follow us on: LinkedIn | Instagram | Twitter | Facebook Our commitment to Diversity, Equity, and Inclusion At NIQ, we are steadfast in our commitment to fostering an inclusive workplace that mirrors the rich diversity of the communities and markets we serve. We believe that embracing a wide range of perspectives drives innovation and excellence. All employment decisions at NIQ are made without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, marital status, veteran status, or any other characteristic protected by applicable laws. We invite individuals who share our dedication to inclusivity and equity to join us in making a meaningful impact. To learn more about our ongoing efforts in diversity and inclusion, please visit the https://nielseniq.com/global/en/news-center/diversity-inclusion
Kurang cocok? Lowongan serupa yang sedang buka
Lihat 5 lowongan serupa →Hanya untuk pelamar di luar Indonesia
Jelajahi lebih lanjut
Data & laporan pasar
Riset gaji & permintaan skill dari data lowongan kami sendiri.
- Skill yang Dicari Perusahaan Remote di Era AISkill paling sering disebut di lowongan aktif per jenis, skill AI, dan yang tumbuh tercepat. Data mingguan.
- Lowongan IT Indonesia vs Remote Global (2026)Analisis data primer 2.049 lowongan: metodologi, klasifikasi, dataset bisa diunduh.
- Permintaan Skill AI: Indonesia vs Global (2026)10.000+ lowongan, classifier taxonomy-first, Wilson CI, pra-registrasi sebelum analisis.
- Remote ≠ Remote: Skill yang Membuka Kerja Global untuk Indonesia (2026)12.891 lowongan remote: skill coding bergaji tertinggi justru paling terkunci untuk pelamar Indonesia. Dataset agregat CC BY 4.0.
- Lapisan Kepatuhan dalam AI Hiring Stack (2026)6.349 lowongan remote: 77,2% tidak menyebut siapa yang boleh melamar. Metodologi dan dataset agregat CC BY 4.0.
- AI di Iklan Lowongan: Indonesia vs Global (2026)Iklan engineering global yang menyebut AI naik dari 3,4% ke 23% sejak ChatGPT; iklan Indonesia dari 2,2% ke 2,8%. Skenario 2027 dan dataset CC BY 4.0.
- Indeks Masa Depan Pekerjaan: semua edisi kuartalanSatu indeks per kuartal dari iklan lowongan remote global dan Indonesia. Q4 2026 adalah baseline (= 100); edisi berikutnya diukur terhadapnya pada sumber yang sama.
- Indeks Masa Depan Pekerjaan, Q3 2026 (pra-baseline)Iklan lowongan remote global dan Indonesia per keluarga pekerjaan: permintaan keahlian AI, jangkauan remote, keterbukaan gaji dan senioritas. Setiap angka dengan ukuran sampel. Dataset CC BY 4.0.
- Resesi Pekerjaan Laptop: Coder Indonesia dan Iklan Lowongannya (2026)Coder Indonesia bertambah delapan kali lipat, iklan lowongan software turun dua pertiga. 284.000 iklan arsip Asia Tenggara 2019–2026, grafik per keluarga pekerjaan, dan taruhan 2027. Dataset CC BY 4.0.
- Pandora's Box Ep. 1: AI dan Iklan Lowongan Asia Tenggara (2026)118.545 iklan lowongan arsip 2019–2026: iklan web dan software developer menyusut di semua papan, aplikasi ponsel jadi tiket masuk, dan Job X-ray untuk 17 pekerjaan. Dataset CC BY 4.0.
- Cron vs LLM vs LangGraph vs Agen untuk Posting Sosial (2026)Studi pra-registrasi empat cara mengubah artikel blog menjadi posting sosial, di dua model. Pemeriksaan otomatis memilih template cron; model lebih menentukan daripada framework.
- Laporan Hiring Indonesia: Tech vs Non-TechPermintaan lowongan per bidang dari hitungan agregat — bukan listing per-listing.
- Benchmark Gaji IndonesiaKisaran gaji agregat lintas peran, dengan metodologi dan dataset terbuka.
- Indeks Gaji & Permintaan Kerja Remote untuk IndonesiaBerapa banyak lowongan remote global yang terbuka untuk Indonesia, dan gajinya (USD) per bidang.
- Laporan Kuartalan Pasar Kerja IndonesiaPHK, pendanaan, gaji & skill per kuartal — agregat terbuka.
- Laporan Pasar Remote per PeranLaporan otomatis per kelompok peran — skill, senioritas, perusahaan, gaji.
- Benchmark Gaji Remote GlobalGaji tahunan per bidang & mata uang, plus porsi lowongan terbuka untuk seluruh dunia.
Perusahaan ini tampaknya hanya merekrut di wilayah tertentu.