Find Jobs
Hire Freelancers

Postgres Query Generation over Millions of Rows of Oil and Gas Data (XPATH a Plus)

$750-1500 USD

Completed
Posted over 6 years ago

$750-1500 USD

Paid on delivery
Project Description is too long for this post. Please read attached word doc for full description. We would like to buy a block of 80 hours doing PostgreSQL queries over Oil & Gas data. We are hovering at around 200 million rows of data that feed multiple user interfaces. All of this data needs a ton of cleaning and curation. The PostgreSQL stuff will not be hard to do. Lot of case when, concat, substrings… just moving data from poor formats to good formats. For example, we may get a VARCHAR that is 1/1/2017 and need to convert it to date. Two second change. But we have hundreds of these date fields. Sometimes in a table with 1MM rows, we will have one badly formatted date – 1&1/2017. Getting that one bad date fixed is not the goal. We have to work around it to get the other 999,999 into date format. We need that VARCHAR in date format. And sometimes, the dates come in as 1/2017. Or some come in as MM/YYYY and others (in same field) are in MM/DD/YYYY format. You get the picture! The easy part will be PostgreSQL. The hard part? Oil & Gas data is not written in lay speak. It’s a very technical industry… tubing, casing, tvd, formations… little of it will make sense. Of course, if you know O&G, huge bonus, but it’s not necessary or expected for this project. We will interpret all the O&G stuff (for you) and send you simple sentences on what each c_ and m_ field should look like. You will then build the queries for each table. 80 hours could just be the beginning. We are already working on a Phase II. We would like to build aggregation queries around all 200 million rows for “highlights” we can report on a webpage. This would entail running row counts/sums/etc. for a ton of fields and even some basic equations. We would update these queries every night and have a web page that reports all the “macro” changes to our data sets each day. This would be used to educate users on how vast and complex the data is behind the UIs. Finally, if you know XPATH, that’s a huge bonus for this project. We can engage you beyond the 80+ hours of PostgreSQL work to running our Web Scraper to go get more data. More data? More PostgreSQL cleaning and curation work. XPATH is not mandatory though. We prefer someone who is super comfortable writing queries. You will need that comfort while you are working with the Oil & Gas terminology.
Project ID: 15634240

About the project

17 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Hi There, This is Nico, I was invited by Stella to the project. Already took a look at the attached document and even I´m more proficient working with MySQL, at the end of the day, and based on what you wrote, it is about writing proper SQL queries (which I know I´m very good at!). I will be happy to have small test job (for free of course) in case you want to see if this fits for both of us. Also, in the other hand, I worked a lot with xPath. The entire configuration system of my personal php framework relies on xPath queries. Thanks for reading my proposal and Cheers, Nico
$1,764 USD in 30 days
5.0 (3 reviews)
5.2
5.2
17 freelancers are bidding on average $1,328 USD for this job
User Avatar
Greetings. I just have a visual over your project's details and feel privileged to tell you that this correlates with my domain of expertise. Please message me as i will like to ask a few questions. Relevant Skills and Experience I have a lot to disclose as of my skills and looking forward to show all of it on chat. Proposed Milestones $1250 USD - default Please provide all requirements (Technical, Non-Technical and business logic).
$1,250 USD in 20 days
5.0 (3 reviews)
5.2
5.2
User Avatar
Hi, how are you today? I read the description of your request, I get the complexity of the task. Cleaning and Querying records from a database can be a demanding task based on how the dirty the data is. I have worked with various industries (banks, leasing institutions, insurance companies) with huge financial and clinical records kept in an inconsistent manner and had to clean and restructure it. So I fully understand your point and I can provide you with the service you require. I am also a new member here, so be rest assured that I will give my best to the project. You can contact me here or via skype(olarid29) to discuss the details
$800 USD in 22 days
4.9 (12 reviews)
4.0
4.0
User Avatar
Hello, I would like to work in your project. I have two years of experience in portgresql with complex queries. For me it would be a real challenge to help them. you can write me i am allways here. Habilidades y experiencia relevante sql ,postgresql, functions, query, triggers Htos propuestos $1250 USD - complete work
$1,250 USD in 10 days
4.6 (5 reviews)
3.1
3.1
User Avatar
The project sounds quite interesting and a long one. The problem area is cleaning so many-many rows perfectly. Better to handle it using stored procedures and some sql jobs. for future have such jobs. Relevant Skills and Experience SQL, MySQL, New to PostgreSQL but confident on writing SQL. Proposed Milestones $1444 USD - On completion Hope to discuss in details on the step by step process to follow
$1,444 USD in 30 days
5.0 (3 reviews)
2.9
2.9
User Avatar
Hi there I understand that you need an expert in database, specially PostgreSQL. I finished a telecom audit project in postgres which needed about 100TB space containing all call detail record of everyday call. It needed to do a lot of analysis, I understand how much details is required to do so. I am also a DBA on SQL Server since 2001. So, for the database part you can rely on me 100. I use selenium, java and xpath on regular basis, so that is not a problem at all. The "hard part" is domain knowledge on Oil and gas industries, I am not an expert on that, however being a Mechanical engineer I believe that I will be able to interpret the terms easily if you send some reference. Let me keep this concise as you may be flooded with many proposals. Please initiate chat for further discussions. Best regards Munir
$2,000 USD in 10 days
5.0 (5 reviews)
2.2
2.2
User Avatar
I have 13 years experience in development so complex large queries are not uncommon to me. I agree I have no knowledge in oil and gas domain, but if I get proper explanation of the tables I can work Relevant Skills and Experience SQL Proposed Milestones $333 USD - Start Task $1000 USD - Complete Task I am waiting for a discussion.
$1,333 USD in 10 days
5.0 (2 reviews)
1.9
1.9
User Avatar
I guarantee you a first class service with the quality and in the time it deserves by putting at your disposal all my knowledge and years of practice. Habilidades y experiencia relevante I have more than 6 years of experience in the role of data architecture, working theoretically and practically with the concepts proposed and others such as: replication, migration, ETL, big data, etc Htos propuestos $100 USD - Determination and breakdown of the requirements to be developed $1100 USD - Development and materialization of the requirements $44 USD - Validation of deliverables with the client Servicios adicionales ofrecidos $1 USD - Optimize, migrate, partition and perform ETL processes. $1 USD - Clustering, sharding, replication-set, security and data services $1 USD - Encryption, data model design, aggregation and indexing. $1 USD - Design and construction Relation and NoSQL Database $1 USD - Development of functions, triggers, internal processes and functions to catalog.
$1,244 USD in 18 days
5.0 (2 reviews)
1.0
1.0
User Avatar
I'd like participate the project. I have enough free time and good relevant experience to do this. My working hours are within 6AM-8PM UTC, so have normal intersection with USA business time. Relevant Skills and Experience Briefly about me: Mid Java dev with more than 11 years in software development. I'm good in databases (was a team member of MS SQL Srvr 2005, now mostly work with Postgres). Proposed Milestones $800 USD - 40 hours spent $533 USD - Done Looks I got the point with this job, but I'd like ask you fo additional chat/call for details.
$1,333 USD in 20 days
5.0 (1 review)
0.5
0.5
User Avatar
Hello Thank you for your posting job. I have checked your job in detail and am confident on your job. I have full experience in IOS/ANDROID app development and ever made a lots of mobile apps Relevant Skills and Experience Database Administration, PostgreSQL, XML Proposed Milestones $1250 USD - a
$1,250 USD in 20 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi, I work with PostgreSQL on a daily basis, creating reports for the different areas of the company that I work for. I'll be glad to help you cleaning your data. Jonathan.- Relevant Skills and Experience I have over 3 years of experience working with PostgreSQL, making reports from millions of rows and several tables that are not on the same server. Proposed Milestones $200 USD - Knowing the data. $800 USD - Planning the query's according to how badly formatted is the data. $300 USD - Execute the query's and look at the data again. $144 USD - Clean up what has not been contemplated, if needed. Is the database constantly getting this bad formatted data ? If so, we could create control query's.
$1,444 USD in 25 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hi there, Hope you’re doing well. I’m interested in the job and would like to work on it over 80+ hours or more. I have around 10 years of SQL experiences and also have sound knowledge of Postgres SQL, as well as XPATH. I have confidence that I can get the job well done with good quality. Feel free to contact me for more details if any. Look forward to hearing from you soon. Best Regards, VL
$1,111 USD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I have worked as a SQL (relational) database consultant for over 25 years. Also, I have well over seven years using PostgreSQL and GIS data (the majority of my PostgreSQL experience includes using the PostGIS spatial extension). I am familiar with SQLite, Spatialite, MySQL and more. I can transform, index appropriately, provide tuning down to operating system level and deploy just about any kind of data (XML, GIS, composites and C constructs...) and also deploy locally using a variety of webservers, proxies and geoservers/mailservers etc. I have worked in the utilities sector, and have recent contract XML experience. Relevant Skills and Experience I started my career as a utilities consultant. One of my clients, EE, was the only compliant provider following UK deregulation. I used XML principally (at AVOX UK) as a freelance Oracle consultant. Proposed Milestones $250 USD - Requirements gathering, documentation and scope of work. Initial sign-off. $500 USD - First round of data cleansing (full ETL) and reporting. Performant testing schemas set up. $250 USD - Second round of data cleansing (full ETL/reports) following feedback and sign-off of first round. $111 USD - Full documentation, handover and any training. Final sign-off to client satisfaction. Additional Services Offered $500 USD - Ad-hoc ongoing development, support, maintenance, further deployment and upgrades. Any ETL tool preference, ETL being CSV->c_*->m_* respectively? psql scripts? kettle/talend etc.?
$1,111 USD in 20 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Oklahoma City, United States
5.0
16
Payment method verified
Member since Aug 16, 2017

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.