ETL/data engineering for Python web app

$25-50 USD / hour

Closed

Posted

almost 2 years ago

$25-50 USD / hour

Looking to have a pipeline to retrieve data via API call from data source, transform the data, and load into BigQuery on GCP. What we are doing: Building a web app that combines features (ie. data fields) from many different sources for many different records. For example, there are over 600 records and each record has tens of data fields from various data sources. The data will be combined into a single, rank-ordered list based on user input. For example, there will be a static list of features (ie. data fields) from various sources (eg. US Census, FBI crime data, NOAA weather, etc) and the user will fill out a questionnaire. Based on the input from the questionnaire, the static data will be scored and combined into a single number. What we need from you: - Write the code to data from source via API call for each record - Adjust the data (ie. clean, impute missing, and transform) - Load the data into BigQuery This pipeline will be run one time per year. However, the pipeline will need to pull a lot of data (5-15 features for more than 600 records). This projects is to build a pipeline for a single source. Based on the quality that is delivered, additional pipelines for other sources can be contracted. The pipeline needs to written in Python and must be deployed on GCP.

Python

API

Google Cloud Platform

ETL

Software Development

Project ID: 34044723

About the project

57 proposals

Remote project

Active 2 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

57 freelancers are bidding on average $38 USD/hour for this job

@AwaisChaudhry

Hi there! I have read what you exactly need and I have great experience with Software Development, ETL, Google Cloud Platform, Python and API. However I would like to ask you a few questions. I do work smart and do not rest until I get the job done. Please feel free to ping me anytime so we can have a detailed discussion. If I can deliver I will deliver in best possible way. Thanks My portfolio: https://www.freelancer.com/u/AwaisChaudhry?w=f

$50 USD in 21 days

5.0

(16 reviews)

6.8

@saubhagyamweb

* * * * * * * * * * * * * * * * * * * * * * Hello Joe , I’m a Professional and Expert Python Developer. I am interested in doing this job for you.I do have the necessary skill set & resources to complete this project. I have done this kind of project before and I am ready to start the project asap.I will provide you with a project covering all the requirements that you are asking for. For the rest of the details, how about connecting on a Chat Box? Awaiting your ping. Thank you Virang Patel

$25 USD in 40 days

4.9

(35 reviews)

6.6

@gogora

Hi That is quite easy task for now. For start please specify the APIs (How many there are, what documentation they have etc.). I could deliver a single source pipeline for the start and we can continue from there. Please discuss details over chat. Maximum response time is 12 hours, although mostly I respond in minutes. Thanks for your attention Archil

$30 USD in 30 days

5.0

(5 reviews)

5.5

@oleksdevr

Hi . I am a Senior Python Data Processing developer. I've completed lots of similar projects using python. I will provide high-quality results in time. I'd like to know more about the project. I can start your project soon. Thank you.

$35 USD in 40 days

4.6

(14 reviews)

5.6

@sapnathakur14

Hello, How are you? I read your job post, and I am interested to work with you on this project, as I have relevant skills set, you can check my profile for your surety. Rest we will finalize everything after complete discussion. Let's hit me message here for further discussion. Thanks and regards Sapna Thakur

$25 USD in 40 days

5.0

(6 reviews)

5.3

@letshappy

Hello I understood your requirements but I have a few doubts, text me so I can clear all my doubts right away

$60 USD in 40 days

4.9

(20 reviews)

5.4

@ognjenstev00

Hi, there. I am an experienced web developer and have previously worked on several GCP applications, so I will provide you quality work according to your instructions. Please contact me so we can discuss in more detail. Thank you.

$38 USD in 40 days

4.9

(4 reviews)

4.9

@cassargammer

Hello there, Thanks for your job posting and I am interested in your project. I have an excellent background as a Data Engineering and Data Science Expert, have rich experiences with ETL, ELT in python. I would like to discuss more details via brief chat. Warm Regards. Cassar

$40 USD in 40 days

5.0

(5 reviews)

4.6

@jaxpeng

Hi there, I’m a Professional and Expert Data engineer. I am interested in doing this job for you. I do have the necessary skill set & resources to complete this project. I have extensive experience in selenium and headless chrome driver on Linux,also have experience with cloud environments like AWS, GCP and ali-yun I am ready to start the project asap. I will provide you project with that covers all the requirements that you are asking for. For the rest of the details, how about connecting on a Chat Box? Awaiting your ping. Thank you

$35 USD in 40 days

5.0

(4 reviews)

4.2

@nenad44556

Hello! Thanks for post project. I checked your project description and I am absolutely sure that I can do the project very well. I am expert in python. I can achieve the results that you are asking for. If you award me, I will do best for you. Please contact me. Thank you

$38 USD in 40 days

5.0

(3 reviews)

3.5

@changshanlife

Hi there, I have experience building ETL pipelines that extract data(query data from SQL DB directly via python scripts, retrieve data via API call or web scrapping from data source) transform the data, and load into BigQuery on GCP. I'd love to chat with you to demonstrate my experience. Regards, Wang

$40 USD in 40 days

5.0

(2 reviews)

3.6

@dataspro

Hello: My name is Anthony Muñoz, I express my interest in working on your project after carefully reading the requirements and concluding that they match my area of knowledge and skills. I am currently the lead engineer for the IT agency DSPro and I have more than 10 years of experience in the field. I have successfully completed a large number of similar jobs on this and other Freelance platforms and I consider your project to be a challenge in which I would like to work and be able to make it a reality. Please feel free to contact me, it will be my pleasure to help you. I greatly appreciate the time provided and I remain attentive to any questions or concerns. Greetings

$35 USD in 40 days

5.0

(1 review)

2.9

@tridhyatech

⭐Hello!⭐ As a senior python developer, I read your project description very carefully and i will deliver all the things as per your requirements. Deliverable by me: - I will write the code to data from source via API call for each record. - Adjust the data as per your requirements (ie. clean, impute missing, and transform). - I will Load the data into BigQuery as per your requirements. I have a deep understanding and experience in the areas of PYTHON that you mentioned. So, I would like to go through more specific discussions with you to provide successful results. Regards, Rahul

$25 USD in 40 days

5.0

(1 review)

3.1

@DidiaSc

Greetings, I'm a professional developer with Bachelor's Degree in Computer Science and more than 10 years of Web development experience. I started my programming career after graduating university mainly with C/C++ and Python - thus I have dealt very much with the algorithms and SDLC workflow. I am very well-versed in agile approach in project development and really love to add my creativity to the overall project implementation. I'm familiar with python beautifulsoup, scrapy and selenium and developed a lot of scraping applications and implemented to website. Also I know python django, flask and odoo ERP system. Also I know ML modules, algorithms and systems like openCV, pandas, keras, tensorflow, cnn, knn, dnn... Ready to discuss more about the project during the interview. Best regards,

$38 USD in 40 days

5.0

(1 review)

3.1

@omemajawaid

The pipeline needs to written in Python and must be deployed on GCP. I am a professional in python Programming and Django Development with more than 9+ years of experience with web app development. Services I provide are: • Development of Restful APIS with Django • Integration with third parties APIS • Payment Gateways(Stripe, Paypal) Integration for your website • SMS/Email Integration (Twilio, Mailgun) for User account recovery and notifications • Custom Database Integration • Deploying your Django Projects on Aws, Heroku, Digitial Ocean,Azure or any cloud service. Feel free to ask me if you have any queries or questions would like to consultant with your requirements with my experience.

$25 USD in 20 days

5.0

(1 review)

2.4

@jjsanchezb

Hi. I have 3 years of experience in python data processing and cloud management. I'd like to get more details about your requirements thanks

$25 USD in 40 days

5.0

(2 reviews)

1.8

@jamesohoff

Hello I have read your project detail. I'm very interested in your project. I am a senior engineer with 5 years of experience in Python, BigQuery, Pipeline processing, GCP. I can work in full-time and deliver quality codes to tight deadlines. I’m interested to hear more about the project from client. If you award me the project, I’d be pleased to discuss this further and get started for you as soon as possible. I hope I'm hearing good news from you soon. Thanks Best regards.

$35 USD in 40 days

5.0

(1 review)

0.8

@TMBS

Hi I Hope you have been good, I have read your requirement, and we find ourselves the best fit and would like to work on this project of yours. Please let me know your available time to discuss requirements & details so I can provide you with a detailed proposal. We will also discuss the scope of work, and timeline at the discussion. We TMedia Business Solution Pvt Ltd have our presence in the US, Canada & India and are available for 24*7 support. I'd love to hear from you for our possible future cooperation.

$30 USD in 40 days

5.0

(1 review)

0.9

@vladyslavmazurok

Hello! I consider my self perfect match for this job because project skill sets are very suitable for me. I have a profound knowledge in programming skills and languages such as API, ETL, Google Cloud Platform, Software Development and Python. Again my work have always been appreciated by my clients because of my sleek communication and smooth handling of projects. Please come to chat box for detail discussion on project.

$35 USD in 31 days

0.0

(0 reviews)

0.0

@ayesha0124

Hi there, I believe I can help you with this project as i have the experience with API, Software Development, ETL, Google Cloud Platform and Python. here is my portfolio: https://www.freelancer.com/u/ayesha0124

$50 USD in 26 days