1
Cluster Computing and MapReduce Lecture 3
Cluster Computing and MapReduce Lecture 3
DATE: 2007/08/28::
2
Google File System (GFS o GooFS)
Google File System (GFS o GooFS)
DATE: 2012/05/11::
3
Google File System - parte 1
Google File System - parte 1
DATE: 2013/08/16::
4
Hadoop Tutorial: Intro to HDFS
Hadoop Tutorial: Intro to HDFS
DATE: 2012/10/31::
5
El Google File System
El Google File System
DATE: 2012/05/30::
6
Google File System - parte 2
Google File System - parte 2
DATE: 2013/08/16::
7
The Linux File System - Explained
The Linux File System - Explained
DATE: 2012/11/17::
8
Google Drive SDK Live: Google Drive vs Traditional File Systems
Google Drive SDK Live: Google Drive vs Traditional File Systems
DATE: 2012/08/13::
9
Google Glass Android processes and file system
Google Glass Android processes and file system
DATE: 2013/05/02::
10
Google File System - parte 3
Google File System - parte 3
DATE: 2013/08/16::
11
Help for hacked sites: File system damage assessment
Help for hacked sites: File system damage assessment
DATE: 2013/03/12::
12
Optimal Flash Provisioning for Cloud Storage Workloads in Google
Optimal Flash Provisioning for Cloud Storage Workloads in Google's Colossus File System
DATE: 2014/06/23::
13
Sync File System API
Sync File System API
DATE: 2014/10/14::
14
Chrome Apps Office Hours: Synchronized File System
Chrome Apps Office Hours: Synchronized File System
DATE: 2013/03/13::
15
Lustre File System
Lustre File System
DATE: 2014/04/19::
16
Wuala - a distributed file system
Wuala - a distributed file system
DATE: 2007/11/02::
17
The Reiser4 Filesystem
The Reiser4 Filesystem
DATE: 2007/10/08::
18
Google Android Training File System  7/18
Google Android Training File System 7/18
DATE: 2010/12/22::
19
Basics of Hadoop Distributed File System (HDFS)
Basics of Hadoop Distributed File System (HDFS)
DATE: 2014/08/29::
20
The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing
The XSEDE Global Federated File System (GFFS) - Breaking Down Barriers to Secure Resource Sharing
DATE: 2014/04/03::
21
Sequoia
Sequoia's 55PB Lustre+ZFS Filesystem
DATE: 2012/04/24::
22
The File is Too Large For The Destination File System
The File is Too Large For The Destination File System
DATE: 2012/01/05::
23
Lustre File System presentation, November 2007
Lustre File System presentation, November 2007
DATE: 2013/07/14::
24
Use Google Drive To Edit The Same File in Multiple Places
Use Google Drive To Edit The Same File in Multiple Places
DATE: 2013/05/30::
25
Linux File Systems in the Cloud @ Linux Collaboration Summit 2011
Linux File Systems in the Cloud @ Linux Collaboration Summit 2011
DATE: 2011/05/05::
26
UNDERSTANDING FILE STRUCTURE - File System Concepts (Part 1)
UNDERSTANDING FILE STRUCTURE - File System Concepts (Part 1)
DATE: 2014/09/20::
27
Using the Google File Manager
Using the Google File Manager
DATE: 2009/12/08::
28
Libgdx [Desktop/Android/iOS] Made Easy Tutorial 13 - File System
Libgdx [Desktop/Android/iOS] Made Easy Tutorial 13 - File System
DATE: 2014/05/17::
29
Using your Gmail/Google Docs as file storage part #1
Using your Gmail/Google Docs as file storage part #1
DATE: 2010/09/05::
30
Steve Jobs File System at D3 in 2005
Steve Jobs File System at D3 in 2005
DATE: 2012/06/05::
31
File Server Migration to Google Drive
File Server Migration to Google Drive
DATE: 2012/09/07::
32
exFat vs NTFS File System Comparison Benchmark for Flash Drives - PCWizKid
exFat vs NTFS File System Comparison Benchmark for Flash Drives - PCWizKid
DATE: 2013/03/15::
33
Formatting a Hard Drive into the Fat32 File System
Formatting a Hard Drive into the Fat32 File System
DATE: 2012/10/14::
34
06-GNU Linux (File system Hierarchy Standard FHS) By Eng-Mohamed Bahaa El-Din | Arabic
06-GNU Linux (File system Hierarchy Standard FHS) By Eng-Mohamed Bahaa El-Din | Arabic
DATE: 2014/06/24::
35
File System Encryption on CentOS 6 | Linux Hangout
File System Encryption on CentOS 6 | Linux Hangout
DATE: 2012/01/02::
36
Debootstrap #2 ARM Debian IMG Filesystem for use on devices Android Phones Table
Debootstrap #2 ARM Debian IMG Filesystem for use on devices Android Phones Table
DATE: 2014/05/28::
37
How to fix Google chrome.exe corrupt file error
How to fix Google chrome.exe corrupt file error
DATE: 2010/11/16::
38
Hadoop Tutorial 13 - Limitations of Hadoop File System
Hadoop Tutorial 13 - Limitations of Hadoop File System
DATE: 2013/04/22::
39
How To fix problem "File system errors" in backtrack / Kali linux
How To fix problem "File system errors" in backtrack / Kali linux
DATE: 2014/11/16::
40
What is Google Chrome OS?
What is Google Chrome OS?
DATE: 2009/11/19::
41
Apples iCloud Drive Is Like A New File System For Apple Devices
Apples iCloud Drive Is Like A New File System For Apple Devices
DATE: 2014/08/01::
42
04-RH033 Red Hat Essentials and Fundamentals (Commands FileSystem Basics Lab) By Eng-Mostafa Hamouda
04-RH033 Red Hat Essentials and Fundamentals (Commands FileSystem Basics Lab) By Eng-Mostafa Hamouda
DATE: 2012/01/18::
43
11-CompTIA Linux+ LX0-101 and LX0-102 (Filesystem Hierarchy Standard) By Eng-Wlaa Isam
11-CompTIA Linux+ LX0-101 and LX0-102 (Filesystem Hierarchy Standard) By Eng-Wlaa Isam
DATE: 2013/04/07::
44
Authors@Google: Neil deGrasse Tyson
Authors@Google: Neil deGrasse Tyson
DATE: 2009/04/07::
45
Google I/O 2011: HTML5 Showcase for Web Developers: The Wow and the How
Google I/O 2011: HTML5 Showcase for Web Developers: The Wow and the How
DATE: 2011/05/11::
46
47-Red Hat Enterprise Security Network Services (Network File System (NFS) Lab)
47-Red Hat Enterprise Security Network Services (Network File System (NFS) Lab)
DATE: 2012/01/23::
47
Giraffa File System Demo
Giraffa File System Demo
DATE: 2012/08/04::
48
Export DGN file to Google Earth... cool stuff
Export DGN file to Google Earth... cool stuff
DATE: 2012/02/16::
49
How to Download from Google Docs
How to Download from Google Docs
DATE: 2012/05/10::
50
[ROOT Permissions] How to use "Es file Explorer" inside the Android system!. Removing App
[ROOT Permissions] How to use "Es file Explorer" inside the Android system!. Removing App's.
DATE: 2012/05/22::
NEXT >>
RESULTS [51 .. 101]
From Wikipedia, the free encyclopedia
Jump to: navigation, search
Not to be confused with GmailFS.
Google File System
Operating system Linux kernel
Type distributed file system
License proprietary

Google File System (GFS or GoogleFS) is a proprietary distributed file system developed by Google for its own use.[1] It is designed to provide efficient, reliable access to data using large clusters of commodity hardware. A new version of the Google File System is codenamed Colossus.[2]

Design[edit]

Google File System. Designed for system-to-system interaction, and not for user-to-system interaction. The chunk servers replicate the data automatically.

GFS is enhanced for Google's core data storage and usage needs (primarily the search engine), which can generate enormous amounts of data that needs to be retained;[1] Google File System grew out of an earlier Google effort, "BigFiles", developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.[1] Files are divided into fixed-size chunks of 64 megabytes,[1] similar to clusters or sectors in regular file systems, which are only extremely rarely overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google's computing clusters, dense nodes which consist of cheap "commodity" computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data throughputs, even when it comes at the cost of latency.

A GFS cluster consists of multiple nodes. These nodes are divided into two types: one Master node and a large number of Chunkservers. Each file is divided into fixed-size chunks. Chunkservers store these chunks. Each chunk is assigned a unique 64-bit label by the master node at the time of creation, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated several times throughout the network, with the minimum being three, but even more for files that have high end-in demand or need more redundancy.

The Master server does not usually store the actual chunks, but rather all the metadata associated with the chunks, such as the tables mapping the 64-bit labels to chunk locations and the files they make up, the locations of the copies of the chunks, what processes are reading or writing to a particular chunk, or taking a "snapshot" of the chunk pursuant to replicate it (usually at the instigation of the Master server, when, due to node failures, the number of copies of a chunk has fallen beneath the set number). All this metadata is kept current by the Master server periodically receiving updates from each chunk server ("Heart-beat messages").

Permissions for modifications are handled by a system of time-limited, expiring "leases", where the Master server grants permission to a process for a finite period of time during which no other process will be granted permission by the Master server to modify the chunk. The modifying chunkserver, which is always the primary chunk holder, then propagates the changes to the chunkservers with the backup copies. The changes are not saved until all chunkservers acknowledge, thus guaranteeing the completion and atomicity of the operation.

Programs access the chunks by first querying the Master server for the locations of the desired chunks; if the chunks are not being operated on (i.e. no outstanding leases exist), the Master replies with the locations, and the program then contacts and receives the data from the chunkserver directly (similar to Kazaa and its supernodes).

Unlike most other file systems, GFS is not implemented in the kernel of an operating system, but is instead provided as a userspace library.

Performance[edit]

Deciding from benchmarking results,[3] when used with relatively small number of servers (15), the file system achieves reading performance comparable to that of a single disk (80–100 MB/s), but has a reduced write performance (30 MB/s), and is relatively slow (5 MB/s) in appending data to existing files. (The authors present no results on random seek time.) As the master node is not directly involved in data reading (the data are passed from the chunk server directly to the reading client), the read rate increases significantly with the number of chunk servers, achieving 583 MB/s for 342 nodes. Aggregating a large number of servers also allows big capacity, while it is somewhat reduced by storing data in three independent locations (to provide redundancy).

See also[edit]

References[edit]

  1. ^ a b c d Carr 2006: ‘Despite having published details on technologies like the Google File System, Google has not released the software as open source and shows little interest in selling it. The only way it is available to another enterprise is in embedded form—if you buy a high-end version of the Google Search Appliance, one that is delivered as a rack of servers, you get Google's technology for managing that cluster as part of the package’
  2. ^ "Google's Colossus Makes Search Real-Time by Dumping MapReduce", High Scalability (World Wide Web log), 2010-09-11 .
  3. ^ Ghemawat, Gobioff & Leung 2003.

Bibliography[edit]

External links[edit]

  • "GFS: Evolution on Fast-forward", Queue, ACM .
  • "Google File System Eval, Part I", Storage mojo .
Wikipedia content is licensed under the GFDL License
Powered by YouTube
MASHPEDIA
LEGAL
  • Mashpedia © 2015