Find Jobs
Hire Freelancers

Local Data Management System

$3000-5000 AUD

Closed
Posted almost 6 years ago

$3000-5000 AUD

Paid on delivery
Our company collects various modes of data in the field to conduct condition assessment of critical infrastructure. The data is typically from sensors installed on field robots and includes images, videos, laser scans, environmental data, point measurements etc. The data is also coupled with other miscellaneous attributes such as reports, documents, annotations etc. The data is currently hosted on local servers and is accessible by the employees on their windows/linux/osx platforms using sshfs, nfs or samba. However as the amount of data exponentially grows (currently at 30Tb and expecting to double in the next 6 months) we need to address a few challenges in data management. Current challenges are: - Data stored across multiple server locations and not accessible seamlessly - Difficult to prioritise data between ssd and hdd when requiring different speeds for current projects - No overview of all the data, where it is, when was it last saved etc. - Data is sometime reshuffled in the back end, which means all the code pointing to hard links breaks. - Hard to manage permissions which many users trying to do prototyping and development at the same time - Slow network access on linux machines (could be problem with sshfs) These problems leads to the basic requirements of the project. A. Central repository to store all data A1. A central repository where data can be seamlessly stored. Data is currently spread over a few local machines. This needs to be aggregated such that the end user doesn't have to worry about where the data is actually stored. A2. Set protocols for adding new data (adding done by plugging in drives into the servers or over the network) A3. Data access to be streamlined. Data is currently (and will be) stored on local linux servers. Data to be accessible to users positioned on compute servers (connected to monitors and running linux) or their laptops (running linux, windows or OSX). Usable file paths to data for all linux users needs to be consistent to enable code transfer between machines. The same for windows and OSX machines to enable unified file paths when working on data processing software across multiple machines. A4. Establish correct mounting protocols/scripts needed to allow for maximum read/write speeds. B A barebones dashboard for the data repository (1 developer, 5-10 days including validation and testing) B1. View basic stats on all data (where stored, how to access, last edited, size of data, basic notes etc) B2. Accessible on the network using user login details (if they are an allowed user) B3. Perform storage medium switches - e.g. copy/move data in the backend between ssd and hdd (for speed boosts on short term projects). This should be seamless to all users, who continue to access the data using the same path. B4. Ability to change group owner of the different mount points B5. Ability to see if data is being uploaded to AWS and tag data to be uploaded onto AWS (currently done using cronjob and bash).
Project ID: 16944203

About the project

14 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
14 freelancers are bidding on average $4,748 AUD for this job
User Avatar
Dear,I am Gang Lee . I'm interested in the project you recently posted. I'm a certificated freelancer with over 1000 good reviews from clients. I have experienced knowledge in every part of Web development. I never disappoint my clients and I'm able to lead your project to success and troubleshoot problems. I am LAMP & MEAN stack expert, Asp.Net and have much knowledge in codeigniter Laravel cakePHP, symfony ,YII and CMS(joomla, wordpress). I am sure these will reflect in my past results. I'm very excited to assist you in making your position successful please feel free to contact me directly to discuss this position further. Thank you.
$4,444 AUD in 30 days
4.9 (746 reviews)
9.0
9.0
User Avatar
I am very proficient in c and c++. I have 16 years c++ developing experience now, and have worked for more than 7 years. My work is online game developing, and mainly focus on server side, using c++ under Linux environment. I made many great projects using c++, for example, I made the tools which could convert java codes into c++ scripts, of course garbage collection included, this was very similar to a compiler, and was very complex. I also made our own mobile game using c++, I can show you the demo of client, if you like. I am very proficient in java also. I have a very good review on Freelancer.com, I never miss a project once I accept the job, you can check my review. Trust me, please let expert help you.
$3,000 AUD in 21 days
4.9 (148 reviews)
7.0
7.0
User Avatar
I am having an experience of more than 16 years in this field. I have served as a VP-Technology for on of the biggest B2B company of India. I am an expert of AWS and Linux. I have worked on similar type of project before as well. Have developed infrastructure that hold data up-to 80TB.. Looking forward to discuss the project in details and figure out how both of us can be helpful to each other.
$5,555 AUD in 30 days
4.9 (60 reviews)
6.0
6.0
User Avatar
Hello, I can help with you in your project Local Data Management System. I have more than 5 years of experience in C Programming, C++ Programming, Linux, Network Administration, Python. We have worked on several similar projects before! We have worked on 300+ Projects. Please check the profile reviews. I can deliver your job with in your deadline. Please ping me for more discussion. I can assure the 100% job satisfaction. Thanks,
$5,000 AUD in 30 days
4.9 (41 reviews)
5.8
5.8
User Avatar
Hi ****We dont do Code in Python, we can only assist you with the DB and AWS**** I just have been through your requirement of Local DB Management and AWS assignments and I found myself fit for this job As we are a team of DBA and DBM and also we are a AWS Certified Architect So we'll do all the tasks perfectly you've wrote below I have read your requirements Completely and I am ready to start the work according to you I have huge experience in Development & Designing and my major relevant key skills are : C Programming, C++ Programming, Linux, Network Administration & much more That just makes me perfect fit for this job I have wide range of expertise knowledge in the same technology that your project actually needs. We Prefer to start with some Formal chat. Feel Free to get in Touch Thanks in Advance Saurabh M Tekzee
$4,705 AUD in 30 days
4.9 (15 reviews)
4.1
4.1
User Avatar
Hello, I have more than 23 years of experience in top IT companies based in the USA. I have worked for some of the top IT clients across the globe and I have managed and worked on some of the large and complex large IT projects. 2 years back I have started my own IT venture partnering with some of the best brains in IT industry. We work extensively on AWS, DEVOPS and Cloud based technologies. What we really do. We are a leading provider for solutions in Cloud services, DevOps, Integration and Testing services for the global marketplace. We enables organizations to leverage the years of investment they have made in their application assets, deliver more from IT budgets and reduce the risk of implementing new technologies. We understand our Client's business realities, ambitions and challenges and help them transition to an environment which includes cloud services. We offer array of services based on cloud such as cloud readiness assessment, cloud architecture and design, cloud migration, cloud infrastructure management, cloud compliance and security, container orchestration in cloud, DevOps and automation and testing services in cloud. We help our customers move to appropriate target cloud vendor platform (AWS, Azure, Google, Heroku), deployment option (Public, Private, Hybrid), suitable type of service (IaaS, PaaS, SaaS) and the appropriate level of remediation.  I am looking forward to have a call with you, so we can discuss more in detail and move ahead. I am sure me and my team can provide you quality output and we can develop long term business relations. Thank you
$4,666 AUD in 20 days
5.0 (2 reviews)
3.7
3.7
User Avatar
It is my great honor to apply for this vacant job. I am very hardworking and can work effectively as fast as I can. I have excellent skills in VB, VBA, C++, C#, OPPs, HTML5, CSS3, JS, AJAX, JQYERY, React.js, ANGULAR.JS, BACKBONE.JS, BOOTSTRAP, PHP,Wordpress, ASP.NET, MVC, Excel VBA Web Research and Microsoft Dynamics CRM Version 2011 2013 2015 2016 and I have enough experience with different kinds of tasks as you can see in my "Work History", most of the time I gave my clients a great results. I can work fulltime and will focus to this project. I can follow instructions very well and will deliver the project on time. It is a pleasure if ever you will message me for an interview. I can work as soon as possible.. Ahtesham Khan
$3,529 AUD in 30 days
5.0 (3 reviews)
3.3
3.3
User Avatar
Our dedicated team of software and programming professionals provides our clients with state of the art programming solutions in the python language. Me and my team has 5 years of experience into Python/Django,Selenium & Data Scraping or Web Crawling. Can very well execute this Project and can work at US hours.
$5,023 AUD in 8 days
4.8 (7 reviews)
2.9
2.9
User Avatar
Hi, After reading your description, following conclusions could be developed: 1) We shall introduce Mongo DB to store and distribute big data even if it crosses millions. 2) You already suggested AWS, so Cloud is the best option to keep all data in a centralized system and then could be distributed to the assinged and privilged user types. For this, we would like to create a super admin who distributes roles and responsibilities to various user sets and they view data acordingly. 3) Super admin and user previleges is solving the problem of accessing/ managing/ change or update in data and what point of time and cganging group owners. 4) Data streaming with AWS and storage in Mongo DB and view via Python script is not going to make this system slow anymore. I would definitely like to view the current process, data extraction from sensors and reading data from a hardware so that I may quote in a better way. Regards, Mit-
$5,000 AUD in 60 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of AUSTRALIA
Australia
0.0
0
Member since May 10, 2018

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.