Play Video
1
An MPP database for the cloud using the LAMP stack and Shard-Query
An MPP database for the cloud using the LAMP stack and Shard-Query
::2014/07/28::
Play Video
2
The Basics of Database Sharding
The Basics of Database Sharding
::2013/05/01::
Play Video
3
Surge 2010 ~ Why Some Architects Almost Never Shard Their Applications
Surge 2010 ~ Why Some Architects Almost Never Shard Their Applications
::2011/12/21::
Play Video
4
O
O'Reilly Webcast: How Sharding Works
::2011/02/04::
Play Video
5
What is Database Sharding Defined in Under 20 Seconds
What is Database Sharding Defined in Under 20 Seconds
::2012/09/28::
Play Video
6
Sharding Part 1 of 7 - Introduction
Sharding Part 1 of 7 - Introduction
::2010/11/09::
Play Video
7
Scaling Your Database in the Cloud
Scaling Your Database in the Cloud
::2012/06/20::
Play Video
8
How To Build A Scalable App Without Sharding
How To Build A Scalable App Without Sharding
::2014/03/12::
Play Video
9
Introduction to Shard Query the MPP distributed query engine for MySQL  - PART 1
Introduction to Shard Query the MPP distributed query engine for MySQL - PART 1
::2013/10/14::
Play Video
10
Scaling Rails Applications: Scaling Your Database / Part 2 (Episode #17)
Scaling Rails Applications: Scaling Your Database / Part 2 (Episode #17)
::2013/02/25::
Play Video
11
Two hours, 12 servers, 4 shards and a dapper DBA - all OSS, all the time.
Two hours, 12 servers, 4 shards and a dapper DBA - all OSS, all the time.
::2012/12/12::
Play Video
12
OSCON Data 2011:  Dwight Merriman, "Databases for Agile Development"
OSCON Data 2011: Dwight Merriman, "Databases for Agile Development"
::2011/07/26::
Play Video
13
Building a Scalable Architecture for Web Apps
Building a Scalable Architecture for Web Apps
::2013/02/25::
Play Video
14
Introduction to document database modeling with RavenDB
Introduction to document database modeling with RavenDB
::2014/04/23::
Play Video
15
real-time search infrastructure and architecture at craigslist
real-time search infrastructure and architecture at craigslist
::2014/08/26::
Play Video
16
NoSQL Database Comparison by Ben Engber
NoSQL Database Comparison by Ben Engber
::2013/04/11::
Play Video
17
DPC13: Fun with BIG Data with various databases - David Ennis
DPC13: Fun with BIG Data with various databases - David Ennis
::2013/12/13::
Play Video
18
Typical deployment diagram
Typical deployment diagram
::2013/01/02::
Play Video
19
Horizontal Data Partitioning with HiveDB & Hibernate Shards
Horizontal Data Partitioning with HiveDB & Hibernate Shards
::2012/06/02::
Play Video
20
Stock Trading and Analysis using Oracle NoSQL Database
Stock Trading and Analysis using Oracle NoSQL Database
::2013/07/15::
Play Video
21
Horizontally Scaling Your Database with Django
Horizontally Scaling Your Database with Django
::2012/11/13::
Play Video
22
NoSQL Database Tutorial part3 |.Net C# DAL Application With MongoDB
NoSQL Database Tutorial part3 |.Net C# DAL Application With MongoDB
::2012/08/04::
Play Video
23
Dayz - First Look & Dying From Starvation Like a Noob
Dayz - First Look & Dying From Starvation Like a Noob
::2013/12/26::
Play Video
24
[Day Z] - Our Finest (And Saddest) Moments. Military Loot.
[Day Z] - Our Finest (And Saddest) Moments. Military Loot.
::2014/04/26::
Play Video
25
Building Fast, Scalable Database Applications The Easy Way - VoltDB
Building Fast, Scalable Database Applications The Easy Way - VoltDB
::2012/11/27::
Play Video
26
[Day-Z Commentated Series] The British One, Rifle blast [Part 1]
[Day-Z Commentated Series] The British One, Rifle blast [Part 1]
::2014/04/25::
Play Video
27
Ashutosh Bapat: Scaling out by distributing and replicating data in Postgres XC
Ashutosh Bapat: Scaling out by distributing and replicating data in Postgres XC
::2013/08/25::
Play Video
28
Interview of Steve Francia, Technical Evangelist, 10Gen Mongo DB
Interview of Steve Francia, Technical Evangelist, 10Gen Mongo DB
::2013/04/19::
Play Video
29
Surge 2010 ~ Database Scalability Patterns
Surge 2010 ~ Database Scalability Patterns
::2011/12/21::
Play Video
30
MongoDB Performance Tuning and Load Testing
MongoDB Performance Tuning and Load Testing
::2014/03/28::
Play Video
31
Laws of Sharding for Big Databases and Small Databases
Laws of Sharding for Big Databases and Small Databases
::2012/09/28::
Play Video
32
Discover How ObjectRocket Supercharges MongoDB
Discover How ObjectRocket Supercharges MongoDB
::2014/02/07::
Play Video
33
Shards and HiveDB
Shards and HiveDB
::2008/05/01::
Play Video
34
Database Reliability in Sharding Environment for Big Data Scalability
Database Reliability in Sharding Environment for Big Data Scalability
::2012/09/28::
Play Video
35
Scaling Databases - CS253 Unit 3 - Udacity
Scaling Databases - CS253 Unit 3 - Udacity
::2012/04/25::
Play Video
36
The Use of Column-oriented Databases Simplifies Application Source Code
The Use of Column-oriented Databases Simplifies Application Source Code
::2010/06/03::
Play Video
37
Introducing MaxScale, the new proxy for MySQL/MariaDB
Introducing MaxScale, the new proxy for MySQL/MariaDB
::2014/02/19::
Play Video
38
Cube Drone - Consistent Hashing
Cube Drone - Consistent Hashing
::2014/03/28::
Play Video
39
Jigesh Saheba - MongoDB Days 2013 - theCUBE
Jigesh Saheba - MongoDB Days 2013 - theCUBE
::2013/06/22::
Play Video
40
Automated PostgreSQL Scaling on AWS
Automated PostgreSQL Scaling on AWS
::2013/08/21::
Play Video
41
Lessons Learned: Refactoring a Solr-Based API App, Torsten Koester, smatch.com/ Shopping24
Lessons Learned: Refactoring a Solr-Based API App, Torsten Koester, smatch.com/ Shopping24
::2013/11/11::
Play Video
42
Avoiding Common Traps When Designing an Application
Avoiding Common Traps When Designing an Application
::2013/11/19::
Play Video
43
DrupalCon London 2011: SLICK DATA SHARDING: HOW TO DEVELOP SCALABLE DATA APPLICATIONS
DrupalCon London 2011: SLICK DATA SHARDING: HOW TO DEVELOP SCALABLE DATA APPLICATIONS
::2013/03/22::
Play Video
44
Fragmentation: New Work by Seth Clark
Fragmentation: New Work by Seth Clark
::2013/09/17::
Play Video
45
The Emporis Skyscraper Award 2012
The Emporis Skyscraper Award 2012
::2013/09/17::
Play Video
46
Video 02   MySQL Architecture Part 1
Video 02 MySQL Architecture Part 1
::2014/01/18::
Play Video
47
MySQL and NoSQL at Craigslist
MySQL and NoSQL at Craigslist
::2012/08/29::
Play Video
48
Garantia Data - The in-memory NOSQL Cloud
Garantia Data - The in-memory NOSQL Cloud
::2012/07/23::
Play Video
49
Renormalize - The Sequel to SQL
Renormalize - The Sequel to SQL
::2012/09/12::
Play Video
50
Brahe - Mass scale flexible indexing
Brahe - Mass scale flexible indexing
::2013/05/22::
NEXT >>
RESULTS [51 .. 101]
From Wikipedia, the free encyclopedia
Jump to: navigation, search

A database shard is a horizontal partition of data in a database or search engine. Each individual partition is referred to as a shard or database shard.

Database architecture[edit]

Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than being split into columns (which is what normalization and vertical partitioning do, to differing extents). Each partition forms part of a shard, which may in turn be located on a separate database server or physical location.

There are numerous advantages to this partitioning approach. Since the tables are divided and distributed into multiple servers, the total number of rows in each table in each database is reduced. This reduces index size, which generally improves search performance. A database shard can be placed on separate hardware, and multiple shards can be placed on multiple machines. This enables a distribution of the database over a large number of machines, which means that the database performance can be spread out over multiple machines, greatly improving performance. In addition, if the database shard is based on some real-world segmentation of the data (e.g., European customers v. American customers) then it may be possible to infer the appropriate shard membership easily and automatically, and query only the relevant shard.[1] Disadvantages include :

  • A heavier reliance on the interconnect between servers
  • Increased latency when querying, especially where more than one shard must be searched.
    • Data or indices are often only sharded one way, so that some searches are optimal, and others are slow or impossible.
  • Issues of consistency and durability due to the more complex failure modes of a set of servers, which often result in systems making no guarantees about cross-shard consistency or durability.

In practice, sharding is complex. Although it has been done for a long time by hand-coding (especially where rows have an obvious grouping, as per the example above), this is often inflexible. There is a desire to support sharding automatically, both in terms of adding code support for it, and for identifying candidates to be sharded separately. Consistent hashing is one form of automatic sharding to spread large loads across multiple smaller services and servers.[2][3]

Where distributed computing is used to separate load between multiple servers (either for performance or reliability reasons), a shard approach may also be useful.

Shards compared to horizontal partitioning[edit]

Horizontal partitioning splits one or more tables by row, usually within a single instance of a schema and a database server. It may offer an advantage by reducing index size (and thus search effort) provided that there is some obvious, robust, implicit way to identify in which table a particular row will be found, without first needing to search the index, e.g., the classic example of the 'CustomersEast' and 'CustomersWest' tables, where their zip code already indicates where they will be found.

Sharding goes beyond this: it partitions the problematic table(s) in the same way, but it does this across potentially multiple instances of the schema. The obvious advantage would be that search load for the large partitioned table can now be split across multiple servers (logical or physical), not just multiple indexes on the same logical server.

Splitting shards across multiple isolated instances requires more than simple horizontal partitioning. The hoped-for gains in efficiency would be lost, if querying the database required both instances to be queried, just to retrieve a simple dimension table. Beyond partitioning, sharding thus splits large partitionable tables across the servers, while smaller tables are replicated as complete units.

This is also why sharding is related to a shared nothing architecture—once sharded, each shard can live in a totally separate logical schema instance / physical database server / data center / continent. There is no ongoing need to retain shared access (from between shards) to the other unpartitioned tables in other shards.

This makes replication across multiple servers easy (simple horizontal partitioning does not). It is also useful for worldwide distribution of applications, where communications links between data centers would otherwise be a bottleneck.

There is also a requirement for some notification and replication mechanism between schema instances, so that the unpartitioned tables remain as closely synchronized as the application demands. This is a complex choice in the architecture of sharded systems: approaches range from making these effectively read-only (updates are rare and batched), to dynamically replicated tables (at the cost of reducing some of the distribution benefits of sharding) and many options in between.

Support for shards[edit]

Algolia
Algolia (a search as a service API) supports sharding.
Apache HBase
HBase supports automatic sharding.[4]
CUBRID
CUBRID supports sharding from version 9.0
dbShards
CodeFutures dbShards is a product dedicated to database shards.[5]
ElasticSearch
ElasticSearch enterprise search server provides sharding capabilities.[6]
eXtreme Scale
eXtreme Scale is a cross-process in-memory key/value datastore (a variety of NoSQL datastore). It uses sharding to achieve scalability across processes for both data and MapReduce-style parallel processing.[7]
Hibernate ORM
Hibernate Shards provides support for shards, although there has been little activity since 2007.[8][9]
IBM Informix
IBM supports sharding in Informix since version 12.1 xC1 as part of the MACH11 technology. Informix 12.10 xC2 added full compatibility with MongoDB drivers, allowing the mix of regular relational tables with NoSQL collections, still supporting sharding, failover and ACID properties.[10][11]
MongoDB
MongoDB supports sharding from version 1.6
MySQL Cluster
Auto-Sharding: Database is automatically and transparently partitioned across low cost commodity nodes, allowing scale-out of read and write queries, without requiring changes to the application.[12]
MySQL Fabric is part of MySQL utilities and includes support for sharding. [13]
OrientDB
OrientDB supports sharding from version 1.7
Plugin for Grails
Grails supports sharding using the Grails Sharding Plugin.[14]
Ruby ActiveRecord
Octopus works as a database sharding and replication extension for the ActiveRecord ORM.
ScaleBase's Data Traffic Manager
ScaleBase's Data Traffic Manager is a software product dedicated to automating MySQL database sharding without requiring changes to applications.[15]
Shard Query
Open Source parallel query engine for MySQL. [16]
Solr Search Server
Solr enterprise search server provides sharding capabilities.[17]
Spanner
Spanner is Google's global scale distributed database that shards data across multiple Paxos state machines to scale to "millions of machines across hundreds of datacenters and trillions of database rows".[18]
SQLAlchemy ORM
SQLAlchemy is an object-relational mapper for the Python programming language that provides sharding capabilities.[19]
SQL Azure
Microsoft supported sharding in SQL Azure through "Federations".[20][21] In 2014, support for the under-used Federations was dropped. Although Azure users were using sharding, they were choosing to implement it through application-specific custom approaches, rather than the automatic approach of Federations.[22]

Disadvantages of sharding[edit]

Sharding a database table before it has been optimized locally causes premature complexity. Sharding should be used only when all other options for optimization are inadequate. The introduced complexity of database sharding causes the following potential problems:

  • Increased complexity of SQL - Increased bugs because the developers have to write more complicated SQL to handle sharding logic.
  • Sharding introduces complexity - The sharding software that partitions, balances, coordinates, and ensures integrity can fail.
  • Single point of failure - Corruption of one shard due to network/hardware/systems problems causes failure of the entire table.
  • Failover servers more complex - Failover servers must themselves have copies of the fleets of database shards.
  • Backups more complex - Database backups of the individual shards must be coordinated with the backups of the other shards.
  • Operational complexity added - Adding/removing indexes, adding/deleting columns, modifying the schema becomes much more difficult.

These historical complications of do-it-yourself sharding are now being addressed by independent software vendors who provide autosharding solutions.

See also[edit]

References[edit]

  1. ^ Rahul Roy (July 28, 2008). "Shard - A Database Design". 
  2. ^ Facebook. "Consistent Hashing". 
  3. ^ Ries, Eric. "Sharding for Startups". 
  4. ^ "Apache HBase Sharding". 
  5. ^ "dbShards product overview". 
  6. ^ "Index Shard Allocation". 
  7. ^ http://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1/index.jsp?topic=%2Fcom.ibm.websphere.extremescale.over.doc%2Fcxsovwork.html
  8. ^ "Hibernate Shards". 2/ 8/2007. 
  9. ^ "Hibernate Shards". 
  10. ^ "New Grid queries for Informix". 
  11. ^ "NoSQL support in Informix". 
  12. ^ "MySQL Cluster Features & Benefits". 2012-11-23. 
  13. ^ "MySQL Fabric sharding quick start guide". 
  14. ^ "Grails Sharding Plugin". 
  15. ^ "ScaleBase's Data Traffic Manager product architecture overview". 
  16. ^ "Shard Query". 
  17. ^ "Distributed Search". 
  18. ^ Corbett, James C; Dean, Jeffrey; Epstein, Michael; Fikes, Andrew; Frost, Christopher; Furman, JJ; Ghemawat, Sanjay; Gubarev, Andrey; Heiser, Christopher; Hochschild, Peter; Hsieh, Wilson; Kanthak, Sebastian; Kogan, Eugene; Li, Hongyi; Lloyd, Alexander; Melnik, Sergey; Mwaura, David; Nagle, David; Quinlan, Sean; Rao, Rajesh; Rolig, Lindsay; Saito, Yasushi; Szymaniak, Michal; Taylor, Christopher; Wang, Ruth; Woodford, Dale. "Spanner: Google’s Globally-Distributed Database". Proceedings of OSDI 2012. Google. Retrieved 24 February 2014. 
  19. ^ "Basic example of using the SQLAlchemy Sharding API.". 
  20. ^ "Federations in SQL Azure: Database Solutions with Unlimited Scalability". 
  21. ^ "Federations in SQL Azure". 
  22. ^ "What features or functionality will not be available in Basic, Standard, and Premium?". MSDN. Microsoft. 1 May 2014. 

External links[edit]

Wikipedia content is licensed under the GFDL License
Powered by YouTube
LEGAL
  • Mashpedia © 2014