UFTP High-performance data transfer for UNICORE

Mitglied der Helmholtz-GemeinschaftUFTPHigh-performance data transfer for UNICOREDr. Bernd Schuller, Tim PohlmannFederated Systems and Data divisionJülich Supercomputer CentreForschungszentrum Jülich GmbHJuly 8, 2011UNICORE Summit, Toruń

OutlineFiletransfer in UNICOREUFTP■■■OutlookPrinciplesDeploymentExamplesJuly 8, 2011, UNICORE Summit, Toruń Slide 2

Storages, Jobs and DataHOME, TMP, ...Server-to-serverdata movementHOME, TMP, ...UNICORE ServerLocalFilespaceImport&ExportJobdirectoriesJob datastagingHTTPGridFTPFTP...ClientUNICORE ServerUNICORE ServerJuly 8, 2011, UNICORE Summit, Toruń Slide 3

Data flows using the BFTdata transferBFT import/exportrequires threesocket connectionsOther clientscommandlineclientEclipsebasedclientBFT stagingrequires foursocket connectionsscientific clientsand applicationsGatewayauthentication,firewall transversalUNICORE basic servicesUNICORE basic servicesUNICOREservicesTarget System InterfaceTarget System InterfaceLocal RMS (e.g. Torque, LL, LSF, etc.)Local RMS (e.g. Torque, LL, LSF, etc.)access to data andmanagement ofcomputational jobsJuly 8, 2011, UNICORE Summit, Toruń

Ideal data flowImport/exportwithout extrasocket connectionsOther clientscommandlineclientEclipsebasedclientscientific clientsand applicationsstagingwithout extrasocket connectionsTarget System InterfaceTarget System InterfaceLocal RMS (e.g. Torque, LL, LSF, etc.)Local RMS (e.g. Torque, LL, LSF, etc.)access to data andmanagement ofcomputational jobsJuly 8, 2011, UNICORE Summit, Toruń

Issues with direct data transferFirewall!■■Direct connections from the outside to the TSI login node areusually not allowedStatically opening ports (or worse, port ranges) is a security riskPort opening technique is required■■UDP based hole punching (like Skype), but UDP is not directlysuited for file transferTCP based: passive FTP is widely understood, but consideredinsecureJuly 8, 2011, UNICORE Summit, Toruń Slide 6

Basic idea: use passive FTP to open portsClient1. „PASV“Firewall„5432“2. open5432 forClient3. connect to port 5432ServerFTP portdata port 54324. close control connection5. close5432July 8, 2011, UNICORE Summit, Toruń Slide 7

UFTP deploymentother UNICOREserverscommandlineclientEclipsebasedclientUFTP support sinceversion 6.4.1pseudo FTPlistenUNICORE/XcommandsCMDUFTPDUNICORE/XserverUFTP support sinceversion 6.4.1TSICluster login nodeLocal RMS (e.g. Torque, LL, LSF, etc.)July 8, 2011, UNICORE Summit, Toruń

File transfer using UFTPClientuftp-client1. Createtransfer3. get filetransferproperties (uftp hostand port, # of streams)4. Upload/downloadFirewallUNICORE/X2. inituftpdInit information sent via commandchannel includes:- client IP- file name- authn secret- optional encryption keyFile systemSlide 10

Security challenges and their resolutionuftpd server runs with root privileges (because it needs to access files from allusers)■Switch effective user/group ID before file accessSending commands via the Command channel allows local users toread/write files under any user ID■■Command port not accessible outside the firewallProtect it using client authenticated SSL and ACL fileAttacker might connect to the newly opened sockets on the uftpd server■Client IP is checked, and a secret key is required for authenticationData channels might be sniffed■Optional symmetric encryption (64 bit key, blowfish algorithm)July 8, 2011, UNICORE Summit, Toruń Slide 11

Using UFTP from UCCoptionally add UFTP related preferences …uftp.client.host=localhostuftp.streams=2uftp.encryption=falseSpecify the protocol in file operations>ucc put­file ­s /home/... ­t https://... ­P UFTPSpecify the protocol in your UCC job{Imports: [{ From: “u6://...?protocol=UFTP“, To: … },],}July 8, 2011, UNICORE Summit, Toruń Slide 12

140UFTP performance: example120Transfer rate (MB/sec)10080604020UFTPUFTP encryptedBFT00 50 100 150 200 250 300 350 400File size (MB)Localhost system, Core 2, 4GB memoryUNICORE 6.4.1 with Perl TSIUFTP 1.0.0, 2 streams> ucc put­file ­s /home/... ­t https://... ­P UFTP ­yJuly 8, 2011, UNICORE Summit, Toruń Slide 13

UFTP: a high-performance data transfer for UNICOREUNICORE FTP (yes, think of "Grid- (rather Globus-)FTP“)■ Multiple parallel TCP connections per data transfer■ Client-Server, Server-Server■ Secure, simple, easy to deploy and operate■ Dynamically open ports in the firewall using the „FTP“protocol■ Platform independent (Java) Widely available with UNICORE version 6.4.1■ as rpm, deb, tar.gz thanks to pac(k)manJuly 8, 2011, UNICORE Summit, Toruń Slide 14

OutlookDeployment in realistic environments (DEISA/PRACE and others)URC support (coming soon!)Workflow system support (current version is still 6.3.x)Performance testingEnhancements■■Support multi-homed serversExperiment with buffer sizesOther uses of the „FTP trick“ not related to data transfer? E.g. whensetting up (cloudy, virtual, …) systems dynamically?July 8, 2011, UNICORE Summit, Toruń Slide 15

Thanks for early testing and feedback on UFTP■ Krzysztof Benedyczak■ Michael Rambadt, Michael Stephan, Björn Hagemeier … and thank you for your attention!July 8, 2011, UNICORE Summit, Toruń Slide 16