Closed

Scality S3 project with MongoDB as a metadata database

Scality S3 project with MongoDB as a metadata database

Scality S3 is an open source Amazon S3 server written in Nodejs. It uses LevelDB for storing objects metadata, which would be useful for small projects however it is not reliable for large-scale projects and data might become inconsistent after a system crash or power failure. Also creating backup is not an easy task and is very tricky. The goal of this project is to replace LevelDB with MongoDB.

Scality S3 provided different drivers for handling metadata like bucketfile, in_memory, etc and uses a wrapper to use proper driver according to user configuration. We have to add a MongoDB driver and then set backend metadata to MongoDB in application configuration in order to keep Scality S3 files untouched and add our features.

There is another thing to keep in mind that In S3 API there is a function that lists all objects of a bucket with specified limit. Current implementation loads all objects and filters them with application. Although it uses stream, this would definitely makes problem as data grows and creates performance issues. We need to filter data in database layer by passing required filters and using MongoDB aggregate framework to avoid performance problems.

Another important thing is that Scality S3 has implemented all amazon features like versioning and ACL, and using MongoDB should not lead to ignoring these features or making them work improperly. To do so, we need to use VersioningRequestProcessor which uses WriteCache and WGM classes and we have to re-implement these classes as well.

Finally, the main project modification should be as least as possible in order to keep project updateable and receive latest features and bug fixes.

Skills: node.js, NoSQL Couch & Mongo

See more: project manager database open office, project hotel reception database, prepare project proposal mysql database client, project join mathtype database, computer project show web database msfrontpage, project management access database, quicktime iphone open stream, php copy failed open stream, project management tool database design, project using jquery database, php warning copy failed open stream permission denied, project create music database, php functioncopy failed open stream permission denied

About the Employer:
( 5 reviews ) Frankfurt, Germany

Project ID: #15343311

18 freelancers are bidding on average $2635 for this job

Yknox

Dear. I am a programmer in china - nodejs expert. I have enough experiences to complete your job. I could help you as well. Relevant Skills and Experience nodejs, mongo db Proposed Milestones $2268 USD - complete I More

$2268 USD in 30 days
(83 Reviews)
7.4
stefkev

Hi, I can write a mongodb driver for Scality S3 project, will create optimised queries for filtering data. Will do the versioning and ACL, whatever Scality has created will be ported to use MongoDB Relevant Skills More

$3000 USD in 20 days
(32 Reviews)
6.3
abhilashtv

Key Projects ā“ Project: Ed-Admin+ [url removed, login to view] Ed-admin+ is a browser-based software application which streamlines and simplifies education management, integrating administrative, teaching and learning t More

$3000 USD in 30 days
(9 Reviews)
5.7
tudiptechnology

Hi, I have gone through the project description and we can surely help you out here. We have been developing/maintaining various web applications in NodeJS (both Express and Sails) Relevant Skills and Experience These More

$2000 USD in 45 days
(5 Reviews)
5.9
londonlance

We are a London (Shoreditch) based web development studio. Please get in touch to discuss further. Following are some of our recent projects; [url removed, login to view] A social media post scheduler and manager More

$3060 USD in 30 days
(1 Review)
5.6
OrangeTechsol

Hello, Hope you are doing Great. We are a team of young and enthusiastic Web developer having 5+ years experience,good knowledge and expertise in Java Script,AngularJS,Node Js and MongoDB. Relevant Skills and Exper More

$2533 USD in 30 days
(22 Reviews)
5.4
laxaar

Hi there, Thank you for sharing your requirement. We are a team of technocrats having 8+ years of experience in Web and Mobile design plus development. Talking about our expertise, we possess very good expertise More

$2400 USD in 30 days
(7 Reviews)
5.1
TenStar718

We have full experience in PHP, MySql, HTML5 and Many php frameworks. We have developed school management website, shopping cart and e-commerce project. Relevant Skills and Experience php,mysql,html5,css,javascript ht More

$2368 USD in 30 days
(3 Reviews)
4.9
meet2amitvw

yes i have worked on S3 and other amazon web services can we discuss more in details to understand your requirements in more details? Relevant Skills and Experience i have relevant skills and experience as per your pr More

$2938 USD in 30 days
(1 Review)
4.6
langlangFan

Hi, sir. Thanks for your job posting! I am a senior web developer with +6 years experience and your project is very interesting for me. I will do my best for your job. Relevant Skills and Experience PHP/Laravel/Cod More

$2500 USD in 30 days
(2 Reviews)
4.2
qualifiedcoders

Hi, Good day! We have read your post & would like to discuss as I have expertise in developing applications with complex databases using NodeJs, MongoDb etc. & have delivered over 500+ projects in last few years. More

$2500 USD in 30 days
(2 Reviews)
3.8
songblue

Hi there, I have good experiences on Node.JS, MongoDB and AWS for 3 years up, so I can complete your project as well. Please inform me. Relevant Skills and Experience Node.JS, MongoDB, AWS Proposed Milestones $2500 U More

$2500 USD in 30 days
(2 Reviews)
3.5
MetaoriginLab

We are a Team of Data Scientists having healthy experience into Big Data technologies like Hadoop,Scala and Spark etc. Relevant Skills and Experience The Team has qualified engineers having expertise in solving comple More

$2388 USD in 30 days
(0 Reviews)
0.0
QuantumSatis

Hi, we are open for a new project for right now. Need to clarify as many details as possible in order to provide you with an accurate quote and our vision. Let's discuss. Relevant Skills and Experience Our agency has More

$2500 USD in 30 days
(0 Reviews)
0.0
$2500 USD in 30 days
(0 Reviews)
0.0
SQUAREPIXEL

Hey There, Thanks for posting this project. I just checked the details of the project and this is something which my company can help you with. Relevant Skills and Experience Square Pixel is a company with a team o More

$3000 USD in 45 days
(0 Reviews)
0.0
vrickson

Hi We are AMAZON specialist , we work on : Amazon FLEX Blocks Amason Data Scrapping We are aware of Scality S3 and its features & how it is integrated to Amazon bucket files being implemented. We will repla More

$3055 USD in 30 days
(1 Review)
0.0
freelancerkpis

We are happy to bid on this project. We provides dedicate developers and Development and design services as well. ***** We are having a great team of Mobile developers with 7-8 years of experience Relevant Skills a More

$2916 USD in 30 days
(0 Reviews)
0.0