Challenges, Technology

Getting files over FTPS from ING – what can be easier?

June 11, 2015

Once upon a time, we’ve got a task to obtain some files from ING through FTPS on a regular basis. Those files contained some payment info that was needed to be processed further. Not a problem, right? However, it sounds much easier than it was…

Thus, in short, we need to connect to ING FTPS server, fetch encrypted documents, decrypt them, use S3 copy to S3 bucket and reorder them. So, the work of our uber admin Ilya has started.

FTPS connect

First, we had to connect to FTPS server (please note – not SFTP!) which is run by some unknown Microsoft proprietary software, so we had to use a customised version of LFTP binary. To compile it from the sources we were using LFTP 4.6.1 version and OS Ubuntu 14.04.02 LTS.

By default, it would compile with OpenSSL library support (distro package has GnuTLS library embedded). If not – as in our case: –with-openssl=/usr/lib in configure options.

Do note that GnuTLS version doesn’t work with Microsoft ftps server.

Then we applied the following patches (in order to support Microsoft proprietary server). You can find them all in a trunk (or download up-to-date LFTP version).

m4/ssl.m4

src/lftp_ssl.cc

src/lftp_ssl.cc

src/resource.cc

Now configure and compile:

Thus we can copy LFTP binary wherever we need, say, /opt/bin directory. After that we had to check dynamic library requirements in order to make sure that OpenSSL library is used:

Initialising LFTP session… Please note, that in order to connect to FTPS server you need to have:

  • Your certificate concatenated with your private key (ssl:cert-file option);
  • CA chain, concatenate all intermediate CA certificates in one file (ssl:ca-file option).

Then we simply replicated all files from a remote directory to a local one.

OpenSSL Decrypt and Verify

Now we need to decrypt the documents we fetched from FTPS server. In order to do so, we used a CMS module of OpenSSL library, you can have a closer look at it here – https://www.openssl.org/docs/apps/cms.html.

To decrypt document:

To verify:


where:
server_cert.pem server’s certificate in PEM format.

After we got zipped file – unzip it :).

S3cmd

In the end, we simply copied unzipped documents recursively to a remote S3 bucket using s3cmd and that’s it – hooray – we are finally done!