Remote FS

Dateisysteme

• (L) File system = Dateisysteme
• There are
  • local (NTFS,..)
  • remote (=on another server)
    • NFS, DFS

Goals

Sl.5

GFS (Google File System)

• GFS master
  • has file namespace, which contains the tree with filenames etc., mapping them to chunks
  • Applications/clients talk to master to get info about where to find files, **but not the files themselves! **
  • Files are gotten from the GFS cuhnkservers where they are located
• (D) SPOF = single point of failurer
• Single-master-problem (Sl.8)
• By big chunk size (64mb) you decrease metadtaa/network overhead

Hadoop

Sl.10 ecosystem

Apache HDFS

• not-psix etc., because append-only, but better performance

Blocks

• (D) minimal size that can be written/read
  • The same as Chunks in GFS
• So large because
  • Disk seeks
  • etc.
• HDFS-data werde in Blöcke / Chunks zerlegt
  • basis für Replikation (kein RAID)
    • TODO - how does RAI do it?
  • Not limited by local filesystem
  • wrtie-once-read-many access-model
  • TYPO slide 14 “access’’” -> " ’ access"

Architectur - NameNode

• Verwaltet Namespace-tree and Mapping

• Namespace

  • each has a namespaceID
  • uses inodes for file folders
    • permissions, space quoas, etc
  • Namespace -> list of filenames

• Mapping of blocks to DataNodes (DN)*

  • file.txt -> A,B
    • A -> {DN0, DN1, DN2}
    • B -> {DN0, DN3, DN4} // can be on diff DN too!

• D/ Image (fsimage) = Namespace + Mapping

  • Contained always in RAM of the NameNode

• D/ Persistenter Namespace = Checkpoint (CP)

• D/ Journal / edit log - modification log of the image

  • write-ahead-log (wrt. CP) that contains Dateisystem Änederung

• Journal for fast persistence, then it gets saved “for real” in a Checkpoint

  • After restart, the Image gets recreated from Checkpoint and Journal
  • Journal + Checkpoint are critical components that are usually saved in diff locations

DataNode (DN)

• Identified from a storageID

  • CreatedOnce during first registration

• Registers at aa a NN from storageID + namespaceID

• Block has data+metadata

• DN sends regular block reports and heartbeats (3 secs) to the NN

  • After 10min it’s ‘out of service’ and the blocks are ’nicht verfügbar’

• Gets Instructions from the NN trhough Heartbeat Replies

  • re-registration, shut down, send block report immediately
  • “replicate block N” -> then it gets it from the other DN, not NN!

HDFS Client

• Allows apps to do HDFS things, create/delete/read etc.
• Doesn’t care about replication, asks for blocks
• NN tells it the location of the Datablocks

Checkpoint / backup node

• CP node
  • Exists
  • sometimes downloads checkpoints and journals from NN
  • merges checkpoint and journal and sends the new checkpoint to NN
• Backup node
  • similar but different - read only name node basically - TODO Sl.19

HDFS file operations

Write

Sl.20

• Block-by-block
  • File lease sent to DN1, which then sents it to replica nodes
  • Written only when block-reports from alll nodes come back to NN sehamohsa