THE EXPAND PARALLEL FILE SYSTEM - Arcos

THE EXPAND PARALLEL FILE 

SYSTEM 

A FILE SYSTEM FOR CLUSTER AND 

GRID COMPUTING 

José Daniel García Sánchez 

ARCOS Group – University Carlos III of Madrid

Contents 

2 

The ARCOS Group. 

Expand motivation. 

Expand design. 

Expand evaluation. 

Conclusions. 

Ongoing Work. 

The Expand Parallel File System 

José Daniel García Sánchez – ARCOS Group – University Carlos III of Madrid 

July 2007 - University of Modena

University Carlos III of Madrid 

3 

Founded in 1989 

Three faculties: 

Faculty of Social Sciences 

and Law. 

Faculty of 

Humanities, Documentation 

and Communication. 

Higher Technical School. 




The ARCOS Group 

4 

The Computer Architecture, Communications and 

Systems Group is part of the Department of 

Computer Science. 

20 full time members 

9 PhD’s (2 full professors + 4 associate professors + 3 

visiting professors). 

11 PhD students 




Research lines 

5 

Data management on Grid environments. 

Parallel file systems. 

Optimization of irregular applications. 

OS for Wireless Sensor Networks. 

Real-time systems. 




Some products 

6 

Expand: A parallel file system for cluster and grid 

environment. 

WinPFS: Windows Parallel File System. 

MiMPI: MPI implementation for heterogeneous 

cluster environments 




Contents 

7 





Conclusions. 

Ongoing Work. 




June-93 

April-94 

February-95 

December-95 

October-96 

August-97 

June-98 

April-99 

February-00 

December-00 

October-01 

August-02 

June-03 

April-04 

February-05 

December-05 

October-06 

8 

Trends in the supercomputing 

environment 

500 

450 

400 

350 

300 

250 

200 

150 

100 

50 

0 

Clusters in top500.org 

75 % of supercomputers in top500 are clusters. 





9 

environment 

Number of transistors per chip still doubling every 

1.5 years. 

Does not mean doubling frequency, performance, … 

More space more cores per chip. 





10 

environment 

Grid Computing: Interconnecting supercomputers to 

aggregate geographically distributed resources. 

Applications are deployed somewhere in the grid. 

Applications read input data and produce output data. 





11 

environment 

Clusters becoming the preferred option for 

supercomputing. 

Processors with increasing capacity. 

Grid computing using clusters as a building block. 

I/O will remain as a major bottleneck. 




Storage system typical architecture 

12 

Clients 

Communication network 

Storage network 

I/O server 




Aggregated bandwidth (MB/s) 

Problems with storage architectures 

13 

14 

12 

10 

8 

6 

4 

2 

0 

1 2 4 8 16 

Clients 



NAS 


Solution: Parallelism 

14 

Parallel applications 

Parallel computers 

Exploit parallelism at 

multiple layers 

Parallel file systems 

Parallel devices 




Parallel File System Architecture 

15 

Computing 

node 

Computing 

node 

Computing node 

Apps 

Client 

Communication Network 

I/O 

Server 

I/O 

Server 

I/O 

Server 

File 1 

File 2 





16 

Computing 

node 

Computing 

node 


Apps 

Client 

Communication Network 

GPFS 

I/O 

Server 

I/O 

Server 

I/O 

Server 

Storage Network 

File 1 

File 2 





17 

GPFS 

Computing 

node 

Computing 

node 


Client 

Apps 

Storage Network 

File 1 

File 2 




18 

Expand Parallel File System: 

Motivation 

Provide a high performance storage system by 

using standard protocols and servers. 

Easy integration of heterogeneous systems. 

Reuse and aggregation of existing resources. 

Parallel data access. 




Why Expand? 

19 

A standard data server already includes almost all 

the needed functionality. 

Reuse. 

Standard protocols and servers make resources 

universally available. 

Easy to deploy. 

Independence of the underlying storage 

infrastructure. 

Portability. 




Objective 

20 

Offer a new approach to build PFS for cluster and 

grid environments by using standard data servers. 

 

Advantages: 

No server change needed. 

• Operations at client side. 

Independence of client and server OS’s. 

• Operations through standard protocols. 

Simplified PFS construction. 

• Take advantage of already implemented server high performance 

mechanisms. 

Allows mixing servers with different platforms and OS’s. 

Easy installation and configuration. 




Contents 

21 





Conclusions. 

Ongoing Work. 





Architecture 

22 

POSIX 

MPI-IO 

… 

Expand 

NFS GridFTP RNS-WS Local 

… 

Parallel access 

Server protocol 

Distributed partition 

File 1 

File 2 




File structure 

23 

Expand file: 

Metadata sub-file. 

Several data sub-files. 

Data distributed across 

several servers. 

File-to-server flexible 

mapping policy. 

Sub-files 

Expand file 

0 1 2 3 4 5 6 7 8 

0 

3 

6 

1 

4 

7 

………….. 

Server 1 Server 2 Server 3 

2 

5 

8 




Directory structure 

24 

Logical 

View 

Mapping 

Physical 

View 

Dir1 

/Expand 

Dir2 Dir3 


/export1 /export2 /export3 

Dir1 Dir2 Dir3 Dir1 Dir2 Dir3 Dir1 Dir2 Dir3 

Dir4 

Dir4 

Dir4 

Dir4 

FileA 

FileA 

FileA 

FileA 




Metadata management 

25 

Metadata distributed 

management. 

Two levels. 

Without locking. 

No metadata manager. 


across servers. 

Master node. 

Hashing on name. 

Load balancing. 

Expand file 

0 1 2 3 4 5 6 7 8 ………….. 

0 

3 

6 

block 


1 

4 

7 

metadata 

2 

5 

8 




Metadata management 

26 


management. 

Two levels. 

Without locking. 

No metadata manager. 


across servers. 

Master node. 

Hashing on name. 

Load balancing. 

Expand file 

0 1 2 3 4 5 6 7 8 ………….. 


Metadata 

FileA 

Metadata 

FileC 

block 

Metadata 

FileD 

Metadata 

FileF 

Metadata 

FileB 

Metadata 

FileE 




Parallel access 

27 

read(fd, buffer,count) 

buffer 

Data blocks 

0 1 2 3 4 5 6 7 8 

Expand 

Threads 


0 

3 

6 

1 

4 

7 

2 

5 

8 




MPI-IO interface using ROMIO 

28 

MPI-IO 

ADIO 

Unix NFS PFVS Expand 

IBM 

PIOFS 

SGI XFS 





Dynamic partition reconfiguration 

29 

Server 1 Server 2 Server 3 Server 4 

2 

5 

8 

0 

3 

6 

1 

4 

7 

Instantaneous. 

Deferred. 

hash(file) = server 3 




Dynamic partition reconfiguration 

30 

Server 1 Server 2 Server 3 Server 3 

2 

5 

8 

12 

16 

0 

3 

6 

1 

4 

7 

9 10 

13 14 

11 

15 




Expand cluster versions 

31 

Linux/NFS 

Java/NFS 

NFS 

Server 

NFS 

Server 

NFS 

Server 

NFS 

Server 






Contents 

32 




Expand adaptation for Grid Computing. 


Conclusions. 

Ongoing Work. 




Requirements for a Grid File System 

33 

Hierarchical logical space name. 

Resource Namespace Service (RNS). 

Standard access interface. 

POSIX and MPI-IO. 

Data access. 

GridFTP. 

Security. 

Grid Security Infrastructure (GSI). 

Performance optimization and improvement. 

Paralle I/O. 





Adapting Expand to Grid environments 

34 

POSIX 

MPI-IO 

… 


Expand 



… 

RNS 

NFS NFS NFS 

Internet + GSI 

GridFTP GridFTP GridFTP GridFTP 

Distributed 

partition 

Site 1 Site 2 Site 3 Site 4 





Contents 

35 





Conclusions. 

Ongoing Work. 




Evaluation 

36 

How does Expand behaves compared to other 

existing solutions? 

Cluster 

• PFVS. 

• GPFS. 

Grid 

• Globus Grid services. 




Cluster environment 

37 

8 biprocessors (Pentium VI, 3.2 GHz). 

2 GB RAM per node. 

Network: Gigabit ethernet. 

Expand. 

PVFS. 

GPFS. 




Cluster benchmarnking 

38 

High performance. 

Parallel access to a file: IOR benchmark. 

FLASH I/O benchmark. 

Metadata operations. 

High throughput. 

Image processing. 

Dynamic partition reconfiguration. 




High performance: 

39 

Parallel access to a file 

Parallel program (IOR) making interleaved writes 

and reads to a single file with different access sizes. 

MPI-IO interface 

Process 1 Process 2 

File 0 1 2 3 4 5 6 7 8 




Bandwidth (MB/s) 

40 


Parallel access to a file for writing 

120 

8 processes (writing) 

100 

80 

60 

40 

20 

XPN 

PVFS 

GPFS 

0 

access size 





41 


Parallel access to a file for reading 

140 

8 processes (reading) 

120 

100 

80 

60 

40 

20 

XPN 

PVFS 

GPFS 

0 

access size 





42 



140 

Parallel access – writing (8 KB) 

120 

100 

XPN 

PVFS 

GPFS 

80 

60 

40 

20 

0 

1 2 4 8 16 

Number of processes 





43 

High performance. 


100 

90 

80 

XPN 

PVFS 

GPFS 

Parallel access – writing (256 KB) 

70 

60 

50 

40 

30 

20 

10 

0 

1 2 4 8 16 





High Performance: FLASH-IO 

44 

FLASH is a parallel application simulating 

thermonuclear flashes. 

FLASH-IO simulates I/O operations performed by 

FLASH. 

Data size is proportional to number of running 

processes. 

1 process 73.53 MB 

16 processes 1.16 GB 





High Performance: FLASH-IO 

45 

80 

70 

60 

XPN 

PVFS 

GPFS 

Benchmark FLASH-IO 

50 

40 

30 

20 

10 

0 

1 2 4 8 16 





Files/second 

Metadata: Creating empty files 

46 

1400 

File creation (empty files)) 

1200 

1 process 4 processes 

1000 

800 

600 

400 

200 

0 

XPN PVFS GPFS 

Filesystem 




Files/second 

Metadata: Creating small files 

47 

1200 

File creation (small files) 

1000 

800 

1 process 

4 processes 

600 

400 

200 

0 

XPN PVFS GPFS 

Filesystem 




High throughput 

48 

Parallel application processing a set of 256 images. 

Each process works on a subset of images 

independently. 

No concurrent access to a file. 

Sizes: 

Image file 5 MB. 

Full dataset 2.5 GB. 

The process applies to each image file a fixed 

bitmask to generate a new image file. 




Time (s) 

49 

High throughput: 

Image processing in C 

100 

90 

80 

70 

60 

Image processing (C application) 

XPN 

PVFS 

GPFS 

50 

40 

30 

20 

10 

0 

1 2 4 8 16 





Time (s) 

50 

High throughput: 

Image processing in Java 

450 

Image processing (Java application) 

400 

350 

XPN 

PVFS 

GPFS 

300 

250 

200 

150 

100 

50 

0 

1 2 4 8 16 






Reconstruction time (min) 

51 

Dynamic partition reconfiguration: 

Adding new nodes 

90 

80 

70 

9 

8 

7 

60 

6 

50 

40 

30 

20 

10 

5 

4 

3 

2 

1 


Reconstruction time (min) 

0 

4-5 4-6 4-7 4-8 5-6 5-7 5-8 6-7 6-8 7-8 

Reconstruction Model 

0 




Grid evaluation environment 

52 

Evaluation for high throughput. 

Perform 500 jobs. 

Each job selects randomly a file (among 200) to access. 

File size is 200 MB. 




Testbed environment 

53 

Intel Pentium 4 

3.20GHz 

• 

GridFTP 

Intel Xeon 

Doble Procesador 2.4GHz 

Intel(R) Pentium(R) 4 

CPU 2.40GHz 

• 

GridFTP 

• 

GridFTP 

• 

GridFTP 

Intel Pentium 4 CPU 

2.80GHz 



AMD Athlon 


Evaluated scenarios 

54 

Typical Grid 

Completely transfer 

file to local node. 

Processing starts after 

transfer finishes. 

Globus services for 

transfer. 

globus-url-copy 

Expand 

Direct remote access to 

file. 

No previous transfer 

to node needed! 

The Expand Parallel File System José Daniel García Sánchez – ARCOS 

Group – University Carlos III of Madrid 


Model 1 / Scenario 1 

55 

1 server 

Complete transfer to 

local node. 

Application access 

local copy. 

GridFTP 

Files 




Model 2 / Scenario 1 

56 

4 servers. 

Distributed files. 

Each server stores 50 

files. 

Complete transfer to 

local node. 

Application accesses 

local copy. 


Files 





Scenario 2 (Expand) 

57 

Expand with 1, 2 and 

4 servers. 

POSIX 

MPI-IO 

Expand 

… 


… 

Local node accesses 

remotely needed 

data. 

No previous transfer 

needed. 

RNS 

Internet + GSI 


Site 1 Site 2 Site 3 Site 4 





Time (min) 

Grid Evaluation 

58 

200 

180 

160 

140 

120 

100 

80 

60 

40 

20 

0 

1 site 4 sites 

(distributed 

files) 

Expand (1 

server) 

Expand (2 

servers) 

Expand (4 

servers) 

Model 




Contents 

59 





Conclusions. 

Ongoing Work. 




Conclusions 

60 

It is feasible to build parallel file system by using 

standard protocols and servers. 

Our solution is easily adaptable to different 

environments/situations (cluster and grid are 

examples). 

Performance results are comparable to other 

solutions (even comercial). 




Contents 

61 





Conclusions. 

Ongoing Work. 




Ongoing work 

62 

Add new protocols (e.g. Web Services) 

Evaluation in large clusters and grid environments. 

Use Expand to improve performance when 

accessing replicated data. 




Ongoing work 

63 

Use Expand as intermediate file system in large 

clusters. 

Apps 

Expand 

Cluster File System (PFVS, GPFS, etc.) 

Compute 

nodes 

Parallel 

access 

Network 

I/O nodes 




THE EXPAND PARALLEL FILE 

SYSTEM 

A FILE SYSTEM FOR CLUSTER AND 

GRID COMPUTING 

José Daniel García Sánchez 

ARCOS Group – University Carlos III of Madrid

THE EXPAND PARALLEL FILE SYSTEM - Arcos

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?