File Parser to Scan CSV.GZ Files and Calculate Unique IP Addresses

$10-100 USD

Completed

Posted

about 5 years ago

$10-100 USD

Paid on delivery

Problem: I have a large number of [login to view URL] files that each contain csv data. There are about 250 of these files and each one is about 100mb in size (compressed). The CSVs contain lists of IP addresses in column A and I would like to know how many unique IP addresses there are in the files (total, not per file). Solution: I would like a file parser written in python that will scan the [login to view URL] files and tell me how many unique IP addresses exist within the csv data. By this I mean total unique IP addresses amongst ALL the data files, not just the unique IP addresses within each individual file. Thank you for your assistance. Please include a brief description in your PM along with your bid so I can tell that you actually read the project description instead of using an auto-bidder to bid on the project. Just a sentence or two will do.

Software Architecture

Project ID: 18989668

About the project

15 proposals

Remote project

Active 5 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

Awarded to:

@osmanaydin

Hello sir, I can parse your files for IP adresses as you like. Please take a look at my reviews and portfolio. I am interested in your project as well. I would like to discuss details via pm. I look forward to hearing from you soon. Best Regards,

$100 USD in 1 day

5.0

(43 reviews)

5.9

15 freelancers are bidding on average $68 USD for this job

@kaloyan13

A Bash script will be a better/faster option. Will extract each file, append the IPs to a temporary files, then we will unique sort them, using sort -u -k...

$40 USD in 1 day

4.9

(556 reviews)

7.6

@Mickelson

Hi Nice to meet you. I checked your description. I have some similiar script for processing .gz files. I will use numpy library for your project. Just get total unique count is not problem. My question is how you will share those csv files, so I can test properly. Regards Lian

$100 USD in 10 days

4.9

(119 reviews)

6.9

@namitkhandelwal

Its about your project you posted: things can also be done in Java. We can do all the task you mentioned in here. hello, warm greetings! I am a Java developer working on Java technology since 7+ years having hands on windows and web development experience. I would like to help in your application development. It would be better if you can share more details about it if you are interested. Please message me to discuss more about the requirements. Budget is negotiable. looking for your positive response. Thanks Namit

$70 USD in 10 days

4.9

(81 reviews)

5.8

@AhmadSameer

Hey there, I can develop the CSVs parser to count unique IP addresses. I'm a System Engineer with coding skills. I had developed tons of Python scripts. Would you share more details? Regards.

$100 USD in 10 days

4.7

(29 reviews)

5.4

@ukgolang

Hi, I'm not strong on Python, but could write this for you in Golang and compile it for Windows or Linux. I've done work parsing and analysing BGP router views logs, so have I previous experience working with large amounts of IP data. I can read the data from the decompression stream and use a radix tree for optimum speed and minimum memory usage. I should be able to complete this today.

$55 USD in 1 day

5.0

(6 reviews)

4.3

@VnNorthStar

Hi I checked your requirement, I'm sure I can do it well. Gzip library in Python supplies some methods to manipulate with a gzip big file, read all lines from gzip files. Please give me an opportunity. I will do it perfectly.

$30 USD in 5 days

4.4

(62 reviews)

5.1

@pushpreethanspal

Hey there. Your project looks straightforward. I have done a lot of work with Python and csv files before, and even though I haven't worked with gzipped data, I don't see that being a problem. Even though 100Mb isn't much, I think I'll implement lazy reading so the entire file is not read into memory at once. Let us discuss this further to see if I am the right person for this job.

$100 USD in 7 days

4.9

(4 reviews)

3.4

@edgarizotov

Greetings. There are two ways how we can solve this task: 1) Dump all the data to SQLite DB and fetch all unique IPs via SQL 2) Iterate over every CSV file and add to global variable only unique IPs, but it will be more memory consuming.

$66 USD in 2 days

5.0

(5 reviews)

3.0

@jayantavkumar

I am a database / Business intelligence architect and having more than 14 years of experience in IT industry. I can achieve it using SSIS ETL tool. Let me know if you are ok then we can talk further.

$88 USD in 2 days

5.0

(4 reviews)

3.2

@skyhaubrich

How's it going? No, this is not an auto-bid. I hope this is enough to prove I've read your project description. I'm Sky, a German Development Enginner with more than 6 years experience in Software Engineering. I would like to write the Python script for you. Python offers a great datastructure that already checks if an entry exists or not. This way, we can determine the unique IP addresses. Best Regards, Sky Haubrich

$35 USD in 0 day

0.0

(0 reviews)

0.0

@Nitin216

Have been working in Python for past one year.. Also contributed to an open source org Symy in past Relevant Skills and Experience Previously worked on Python built Sympy(Open Source) Used Python for ML purpose too.

$61 USD in 10 days

0.0

(0 reviews)

0.0

@ramagni

Hi , Hope you are doing well. Should be able to deliver this project in a day max . I understand the requirement . (ip address-is it ipv4 or ipv6 ). I will be using python as language for implementing this. We can discuss further on this , if required My skype I’d @ live:d311a9e099b4554 Thanks, Ram

$66 USD in 1 day

5.0

(1 review)

0.0

@anatole7

Hi. Your task interesting for me. Python is my third language and I look out more practice for it everywhere. I will glad to do it task for you. p.s. my bid is 55 and approximately 1-2 day for work, testing and correction.

$55 USD in 2 days