Cluster Computing and MapReduce Lecture 3

Channel: GoogleDevelopers   |   2007/08/28
Play Video
1
Cluster Computing and MapReduce Lecture 3
Cluster Computing and MapReduce Lecture 3
::2007/08/28::
Play Video
2
Hadoop Tutorial: Intro to HDFS
Hadoop Tutorial: Intro to HDFS
::2012/10/31::
Play Video
3
Google File System - parte 1
Google File System - parte 1
::2013/08/16::
Play Video
4
Google Drive SDK Live: Google Drive vs Traditional File Systems
Google Drive SDK Live: Google Drive vs Traditional File Systems
::2012/08/13::
Play Video
5
Google File System (GFS o GooFS)
Google File System (GFS o GooFS)
::2012/05/11::
Play Video
6
Google File System
Google File System
::2012/06/05::
Play Video
7
The Linux File System - Explained
The Linux File System - Explained
::2012/11/17::
Play Video
8
Wuala - a distributed file system
Wuala - a distributed file system
::2007/11/02::
Play Video
9
El Google File System
El Google File System
::2012/05/30::
Play Video
10
Chrome Apps Office Hours: Synchronized File System
Chrome Apps Office Hours: Synchronized File System
::2013/03/13::
Play Video
11
Google Glass Android processes and file system
Google Glass Android processes and file system
::2013/05/03::
Play Video
12
Google File System - parte 2
Google File System - parte 2
::2013/08/16::
Play Video
13
Google File System - parte 3
Google File System - parte 3
::2013/08/16::
Play Video
14
ZFS - Filesystem einer neuen Generation -  Developer Garden TechTalk (01.12.2011)
ZFS - Filesystem einer neuen Generation - Developer Garden TechTalk (01.12.2011)
::2011/12/19::
Play Video
15
Is NTFS a Journaling File System?
Is NTFS a Journaling File System?
::2010/08/17::
Play Video
16
Distributed File System
Distributed File System
::2012/11/22::
Play Video
17
Tutorial: Global Filesystem mit Panzura
Tutorial: Global Filesystem mit Panzura
::2013/03/26::
Play Video
18
ECSS April 10 2012 Global Federated File System - Andrew Grimshaw
ECSS April 10 2012 Global Federated File System - Andrew Grimshaw
::2012/04/13::
Play Video
19
Google Android Training File System  7/18
Google Android Training File System 7/18
::2010/12/22::
Play Video
20
Help for hacked sites: File system damage assessment
Help for hacked sites: File system damage assessment
::2013/03/12::
Play Video
21
FileSystem 2 de 2
FileSystem 2 de 2
::2013/03/02::
Play Video
22
GFS Global File System
GFS Global File System
::2009/09/08::
Play Video
23
Panzura Delivers Global File System for Enterprise Cloud Storage
Panzura Delivers Global File System for Enterprise Cloud Storage
::2012/07/23::
Play Video
24
How to put files in the file system
How to put files in the file system
::2013/04/27::
Play Video
25
第5回GoogleのChrome担当者だけど何か質問ある?〜FileSystem API特集
第5回GoogleのChrome担当者だけど何か質問ある?〜FileSystem API特集
::2012/05/31::
Play Video
26
Workshop Tecnico 2013 N°3 - ZFS: Zettabyte File System
Workshop Tecnico 2013 N°3 - ZFS: Zettabyte File System
::2013/05/30::
Play Video
27
The Reiser4 Filesystem
The Reiser4 Filesystem
::2007/10/09::
Play Video
28
Rescatux - Filesystem check (Fix forced)
Rescatux - Filesystem check (Fix forced)
::2012/07/07::
Play Video
29
Seattle Conference on Scalability: Lustre File System
Seattle Conference on Scalability: Lustre File System
::2007/10/09::
Play Video
30
27 File System Architecture - 1
27 File System Architecture - 1
::2013/04/10::
Play Video
31
A Brief History of the BSD Fast Filesystem, Kirk McKusick
A Brief History of the BSD Fast Filesystem, Kirk McKusick
::2009/03/05::
Play Video
32
Douglas parte 1 ([    SLIDE    ])
Douglas parte 1 ([ SLIDE ])
::2010/06/11::
Play Video
33
02 Windows 7, A Quck Tour The Desktop
02 Windows 7, A Quck Tour The Desktop
::2009/11/06::
Play Video
34
28 File System Architecture - 2
28 File System Architecture - 2
::2013/04/10::
Play Video
35
Linux Shell - Lesson 1 - File System Navigation
Linux Shell - Lesson 1 - File System Navigation
::2012/07/03::
Play Video
36
How to fix: -Grub rescue: unknown filesystem- Error
How to fix: -Grub rescue: unknown filesystem- Error
::2013/09/19::
Play Video
37
Python without filesystem
Python without filesystem
::2012/07/04::
Play Video
38
29 File System Architecture  - 3
29 File System Architecture - 3
::2013/04/12::
Play Video
39
Moodle 2.0 Repositories File System
Moodle 2.0 Repositories File System
::2010/06/15::
Play Video
40
How To Clean Your File System Free No Downloads! (Windows 8)
How To Clean Your File System Free No Downloads! (Windows 8)
::2014/03/28::
Play Video
41
Web Apps break up with the Internet: Trying out Google Docs offline.
Web Apps break up with the Internet: Trying out Google Docs offline.
::2012/09/09::
Play Video
42
Finger-Ready ACTION FILE System with bonus video on how to set up the 1-31 Day Tickler File System
Finger-Ready ACTION FILE System with bonus video on how to set up the 1-31 Day Tickler File System
::2013/01/07::
Play Video
43
Alfresco - How to Configure CIFS (Common Internet File System) to Replace your Shared Network Drive
Alfresco - How to Configure CIFS (Common Internet File System) to Replace your Shared Network Drive
::2009/05/29::
Play Video
44
Restoring File System Playstation 3
Restoring File System Playstation 3
::2011/12/20::
Play Video
45
Analysis of Filesystem and File Fragmentation in Lustre
Analysis of Filesystem and File Fragmentation in Lustre
::2013/09/20::
Play Video
46
[Linux.conf.au 2013] - Ceph object storage, block storage, file system, replication...
[Linux.conf.au 2013] - Ceph object storage, block storage, file system, replication...
::2013/03/25::
Play Video
47
FileSystem Git
FileSystem Git
::2013/04/14::
Play Video
48
'FUSE'ing Python for Rapid Development of Storage Efficient File-System
::2012/06/12::
Play Video
49
C# Tutorial 73: How to populate TreeView with file system directory structure
C# Tutorial 73: How to populate TreeView with file system directory structure
::2013/06/11::
Play Video
50
Destroying android filesystem
Destroying android filesystem
::2013/05/23::
NEXT >>
RESULTS [51 .. 101]
From Wikipedia, the free encyclopedia
Jump to: navigation, search

Google File System (GFS or GoogleFS) is a proprietary distributed file system developed by Google for its own use.[1] It is designed to provide efficient, reliable access to data using large clusters of commodity hardware. A new version of the Google File System is codenamed Colossus.[2]

Design[edit]

Google File System. Designed for system-to-system interaction, and not for user-to-system interaction. The chunk servers replicate the data automatically.

GFS is enhanced for Google's core data storage and usage needs (primarily the search engine), which can generate enormous amounts of data that needs to be retained;[1] Google File System grew out of an earlier Google effort, "BigFiles", developed by Larry Page and Sergey Brin in the early days of Google, while it was still located in Stanford.[1] Files are divided into fixed-size chunks of 64 megabytes,[1] similar to clusters or sectors in regular file systems, which are only extremely rarely overwritten, or shrunk; files are usually appended to or read. It is also designed and optimized to run on Google's computing clusters, dense nodes which consist of cheap, "commodity" computers, which means precautions must be taken against the high failure rate of individual nodes and the subsequent data loss. Other design decisions select for high data throughputs, even when it comes at the cost of latency.

A GFS cluster consists of multiple nodes. These nodes are divided into two types: one Master node and a large number of Chunkservers. Each file is divided into fixed-size chunks. Chunkservers store these chunks. Each chunk is assigned a unique 64-bit label by the master node at the time of creation, and logical mappings of files to constituent chunks are maintained. Each chunk is replicated several times throughout the network, with the minimum being three, but even more for files that have high end-in demand or need more redundancy.

The Master server doesn't usually store the actual chunks, but rather all the metadata associated with the chunks, such as the tables mapping the 64-bit labels to chunk locations and the files they make up, the locations of the copies of the chunks, what processes are reading or writing to a particular chunk, or taking a "snapshot" of the chunk pursuant to replicate it (usually at the instigation of the Master server, when, due to node failures, the number of copies of a chunk has fallen beneath the set number). All this metadata is kept current by the Master server periodically receiving updates from each chunk server ("Heart-beat messages").

Permissions for modifications are handled by a system of time-limited, expiring "leases", where the Master server grants permission to a process for a finite period of time during which no other process will be granted permission by the Master server to modify the chunk. The modifying chunkserver, which is always the primary chunk holder, then propagates the changes to the chunkservers with the backup copies. The changes are not saved until all chunkservers acknowledge, thus guaranteeing the completion and atomicity of the operation.

Programs access the chunks by first querying the Master server for the locations of the desired chunks; if the chunks are not being operated on (i.e. no outstanding leases exist), the Master replies with the locations, and the program then contacts and receives the data from the chunkserver directly (similar to Kazaa and its supernodes).

Unlike most other file systems, GFS is not implemented in the kernel of an operating system, but is instead provided as a userspace library.

Performance[edit]

Deciding from benchmarking results,[3] when used with relatively small number of servers (15), the file system achieves reading performance comparable to that of a single disk (80–100 MB/s), but has a reduced write performance (30 MB/s), and is relatively slow (5 MB/s) in appending data to existing files. (The authors present no results on random seek time.) As the master node is not directly involved in data reading (the data are passed from the chunk server directly to the reading client), the read rate increases significantly with the number of chunk servers, achieving 583 MB/s for 342 nodes. Aggregating a large number of servers also allows big capacity, while it is somewhat reduced by storing data in three independent locations (to provide redundancy).

See also[edit]

References[edit]

  1. ^ a b c d Carr 2006: ‘Despite having published details on technologies like the Google File System, Google has not released the software as open source and shows little interest in selling it. The only way it is available to another enterprise is in embedded form—if you buy a high-end version of the Google Search Appliance, one that is delivered as a rack of servers, you get Google's technology for managing that cluster as part of the package’
  2. ^ "Google's Colossus Makes Search Real-Time by Dumping MapReduce", High Scalability (World Wide Web log), 2010-09-11 .
  3. ^ Ghemawat, Gobioff & Leung 2003.

Bibliography[edit]

External links[edit]

Wikipedia content is licensed under the GFDL License

Mashpedia enables any individual or company to promote their own Youtube-hosted videos or Youtube Channels, offering a simple and effective plan to get them in front of our engaged audience.

Want to learn more? Please contact us at: hello@mashpedia.com

Powered by YouTube
LEGAL
  • Mashpedia © 2014