Senior Full Stack Engineer
At Daltix (Full-time), in Lisbon, Portugal
Expires at: 2019-12-12
Daltix is a fast-growing, successful, data-driven start-up from Belgium with offices in Boom, Ghent and Lisbon. We are bringing real-time insights to the world of retail. How we are doing that? We have developed a set of tools to gather, process and analyze retail data and to bring this all together in our data platform we offer to our clients. So far, we have had great success in doing so. That's why today Daltix serves customers such as Makro, Lidl, Dreamland, Jumbo, Greenyard,, Unilever and others.
Sign me up! No, wait... What will I be doing?
We are looking for a kickass Distributed System Engineer who is excited to help build and maintain the distributed data collection system that is at the heart of our business. As such, you would have the opportunity to ensure that our data collection engine, which consists of distributed web crawlers, is state of the art and ahead of our competition. You will ensure that we can scrape any webshop, no matter the ban-detection that has been put in place. Next to that, it will be important that the proper monitoring tools are in place. We are currently scraping 60 sites and your goal is to at least triple that without losing completeness and quality.
We are a data-driven company which collects and processes more than 500GB of raw data daily. We leverage big data technologies such as Serverless, Spark on AWS EMR to crunch these volumes of data and make it queryable.
Your typical tasks include the following responsibilities:
Distributed web crawling architectures.
Cost-effective data processing architectures.
Advanced system monitoring solutions & dashboards.
Design advanced ways of interpreting scraped HTMLs.
Advanced proxy management.
Making sure that Daltix stays competitive in terms of data collection by using the latest & most suitable technologies throughout our stack.
About the stack
This distributed system is made on top of Amazon Web Services and uses Serverless architectures where possible, with Python & Javascript being the main programming languages used.
As Daltix scales from 50+ websites to 200+ websites (which it scrapes multiple times per day!), it has to invest in orchestration technologies such as Kubernetes as well as logging & monitoring solutions to keep an overview at scale.
Main requirements
Why you're the one, we are dying to meet:
Required
You have at least 5 years of experience in object-oriented software engineering & design in any object-oriented programming language.
You have experience with and understanding of large-scale web crawling.
You have experience with databases, SQL and infrastructure such as load-balancers, caches, …
You have a high proficiency in spoken and written English.
You never stop learning.
Python
You have expert knowledge of web-scraping & web-scraping architectures.
You have experience with GoLang & JavaScript (NodeJS) is a plus.
You have experience with big data technologies (such as Hadoop, Spark, Airflow, Cassandra, Elasticsearch) is a plus.
You have a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and security.
You get energy from working in a highly complex and challenging startup environment with a high tech product.
Nice to have
Big Plus
You have experience building on top of Amazon Web Services.
You have programming experience with Python
You have expert knowledge of web-scraping & web-scraping architectures.
You have experience with GoLang & JavaScript (NodeJS) is a plus.
You have experience with big data technologies (such as Hadoop, Spark, Airflow, Cassandra, Elasticsearch) is a plus.
You have a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and security.
You get energy from working in a highly complex and challenging startup environment with a high tech product.
Perks
If you still need more convincing…
Flexible work arrangements
We are a young, entrepreneurial and fast-growing company; you will have the unique opportunity to shape our future and have a positive impact on our clients' business
Competitive wage in a talented, international team of top-notch experts
Trainings to stay at the top of your field
A cozy office in the heart of Lisbon
Health Insurance coverage
Fresh fruit, snacks, tea and coffee on the house
Occasional drinks and team events
Besides developing your technical skills you will also have the opportunity to grow into the following skill sets: Technical/architectural lead. SW project management. Team leading & coachin
Apply for this position
---------------------------------------------------------------------------
Visit this link to stop these emails: https://zapier.com/manage/zaps/60675974/stop/?check=0f47aafc8beecbf4542af7172d1633e0
Post a Comment