User:Wuseman/lftp
LFTP is a sophisticated file transfer program supporting a number of network protocols (ftp, http, sftp, fish, torrent). Like BASH, it has job control and uses the readline library for input. It has bookmarks, a built-in mirror command, and can transfer several files in parallel. It was designed with reliability in mind. LFTP is free software, distributed under the GNU GPL license.
Install lftp
root #
echo "gnutls ipv6 nls ssl convert-mozilla-cookies -idn socks5 -verify-file" >> tee /etc/portage/package.use/lftp
root #
emerge --ask lftp
Basic Usage
- Download all folders inside /foo/* on ftp to foo:
user $
lftp -e open ftp://user1:pass1:site1:port -e "mirror -c -P1 remote/path /local/path" -d
- FXP Between SITE 1 and SITE 2 with 20 threads at same time:{
user $
lftp -e "set ftp:use-fxp true ;mirror -R -P20 ftp://user1:pass1:site1:port:/path ftp://user1:pass1:site1:port:/path" -d
- Use lftp to multi-threaded download files from websites:
user $
lftp -c "pget -n 10 http://example.com/foo.bar"
- mirrors directory to a ftp server:
user $
lftp -u 'login,passwd' -e "mirror reverse /my/from/dir/ /ftp/target/dir/" ftp://site.com/
- Mirror a directory structure from websites with an Apache-generated file indexes:
user $
lftp-e "mirror -c" http://example.com/foobar/
- Internet Speed Test:
user $
lftp -e 'pget http://address_to_file; exit;'
- Multi-segment file downloading with lftp:
user $
lftp -u "user,pass" ftp://site.com -e 'pget -c -n 6 file'
- Multi-segment directory downloading with lftp:
user $
lftp -u "user,pass" ftp://site.com/ -e 'mirror -c parallel=3 use-pget -n=5 "Some folder"'
- Gets directory and files tree listing from a FTP-server:
user $
lftp -u "user,pass" ftp://site.com/ -e "du -a;exit" > server-listing.txt
- mirrors directory to a ftp server:
user $
lftp -u login,passwd" -e "mirror reverse /my/from/dir/ /ftp/target/dir/" ftp://site.com/
- Mirror a directory structure from websites with an Apache-generated file indexes:
user $
lftp -e "mirror -c" http://example.com/foobar/
- Internet Speed Test:
user $
lftp-e 'pget http://address_to_file; exit;'
- Multi-segment file downloading with lftp:
user $
lftp -u "user,pass" ftp://site.com -e 'pget -c -n 6 file'
- Multi-segment directory downloading with lftp:
user $
lftp -u "user,pass" ftp://site.com/ -e 'mirror -c parallel=3 use-pget-n=5 "Some folder"'
- Fastest segmented parallel sync of a remote directory over ssh
user $
lftp -u "user,pass" -e "set sftp:connect-program 'ssh -a -x -T -c fooo -o Compression=no'; mirror -v -c loop use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22
Advanced Usage
┌┼───┤ lftp | ftp.sunet.se ├───┤/├─── └┼─$─┤► : lftp -e open ftp://ftp.sunet.se -e "ls -sh about" -rw-r--r-- 1 ftp ftp 2764 Sep 05 2022 ftp-about-SPmkII.html -rw-r--r-- 1 ftp ftp 3297 Mar 29 2019 ftp.sunet.se-history_sv.html -rw-r--r-- 1 ftp ftp 2550 May 07 2007 graph.png -rw-r--r-- 1 ftp ftp 8629 Nov 12 2022 index.html -rw-r--r-- 1 ftp ftp 279 May 07 2007 largefile.dot -rw-r--r-- 1 ftp ftp 2550 May 07 2007 largefile.png -rw-r--r-- 1 ftp ftp 4517 Mar 28 2019 largefile.svg -rw-r--r-- 1 ftp ftp 210 May 07 2007 smallcachedfile.dot -rw-r--r-- 1 ftp ftp 1558 May 07 2007 smallcachedfile.png -rw-r--r-- 1 ftp ftp 3390 Mar 28 2019 smallcachedfile.svg -rw-r--r-- 1 ftp ftp 303 May 07 2007 smalluncachedfile.dot -rw-r--r-- 1 ftp ftp 2545 May 07 2007 smalluncachedfile.png -rw-r--r-- 1 ftp ftp 4682 Mar 28 2019 smalluncachedfile.svg drwxr-xr-x 2 ftp ftp 31 Nov 14 2012 zfs-stats
We can check size as it was our own box and by adding quit to the command we can do things as it was our own shell, after the command has been executed then we disconnect so lets say we want to check size of a ftp dir
lftp -e open "ftp://ftp.sunet.se" -e "du -sh about:quit" 3.6M about
We can fxp files between fxps without downloading to our self, following command will send file from source_server to our destination_server
user $
lftp -e "open -e 'fxp ftp://source_server.com; get /path/to/source/file' ftp://destination_server.com; quit"
We can paralllize a download for both download and fxp, here is a simple example for -c as in continue you can always continue if its broken
user $
lftp -e open -u ${USER}:${PASSWORD} nr1.nu -p 65005 -e "mirror -c -P20 [/path/to/ftp_dir] [/path/to/local_dir]"
It is possible to build really advanced scripts for doing advanced multiple transfers between several sites in parallel, here is an example:
Example script for lftp
# Site 1 SITE1_HOST="site1.example.com" SITE1_USER="username1" SITE1_PASSWORD="password1" SITE1_LOCAL_DIR="/path/to/local/dir" SITE1_REMOTE_DIR="/path/to/remote/dir" # Site 2 SITE2_HOST="site2.example.com" SITE2_USER="username2" SITE2_PASSWORD="password2" SITE2_LOCAL_DIR="/path/to/local/dir" SITE2_REMOTE_DIR="/path/to/remote/dir" # Site 3 SITE3_HOST="site3.example.com" SITE3_USER="username3" SITE3_PASSWORD="password3" SITE3_LOCAL_DIR="/path/to/local/dir" SITE3_REMOTE_DIR="/path/to/remote/dir" # Site 4 SITE4_HOST="site4.example.com" SITE4_USER="username4" SITE4_PASSWORD="password4" SITE4_LOCAL_DIR="/path/to/local/dir" SITE4_REMOTE_DIR="/path/to/remote/dir" # FTP operations - Site 1 lftp -e " open $SITE1_HOST user $SITE1_USER $SITE1_PASSWORD lcd $SITE1_LOCAL_DIR cd $SITE1_REMOTE_DIR # Upload files put file1.txt put file2.txt # Download files get file3.txt get file4.txt # Disconnect and quit bye " # FTP operations - Site 2 lftp -e " open $SITE2_HOST user $SITE2_USER $SITE2_PASSWORD lcd $SITE2_LOCAL_DIR cd $SITE2_REMOTE_DIR # Upload files put file5.txt put file6.txt # Download files get file7.txt get file8.txt # Disconnect and quit bye " # FTP operations - Site 3 lftp -e " open $SITE3_HOST user $SITE3_USER $SITE3_PASSWORD lcd $SITE3_LOCAL_DIR cd $SITE3_REMOTE_DIR # Upload files put file9.txt put file10.txt # Download files get file11.txt get file12.txt # Disconnect and quit bye " # FTP operations - Site 4 lftp -e " open $SITE4_HOST user $SITE4_USER $SITE4_PASSWORD lcd $SITE4_LOCAL_DIR cd $SITE4_REMOTE_DIR # Upload files put file13.txt put file14.txt # Download files get file15.txt get file16.txt # Disconnect and quit bye "
lftp supports SFTP, allowing you to transfer files in parallel without using SCP.
set ftp:use-fxp true set net:limit-total-rate 0:128000 set sftp:auto-confirm yes set sftp:connect-program "ssh -a -x -T -c aes256-ctr -o Compression=no"
During the years I have been using tons of FTP clients for FXPing, for example, pftp and cftp and ftp with parallel and I tried most clients and I must say lftp is the master of the beast. However, it requires a lot of configuration to be properly set up for transferring files between 30-40-50 sites simultaneously. But once it's perfectly configured, I recommend it over anything else.
user $
lftp sftp://username:password@host -e "mirror -R -P20 source target"
Well, there are tons of settings to use with lftp, so the best way to find them all is to read the manual: lftp Manual.
Anonymous Networking
To use lftp with Tor for anonymous and secure FTP connections, you can utilize Tor's SOCKS proxy feature. Here's how you can configure lftp to connect through Tor:
Configure Tor SOCKS proxy: Open the Tor configuration file (/etc/tor/torrc on Linux) and make sure the following lines are present:
/etc/tor/torrc
Configure Tor SOCKS proxySocksPort 9050 SocksPolicy accept *
Start lftp with Tor: Use the following command to start lftp and configure it to use Tor's SOCKS proxy:
user $
lftp -e "set ftp:proxy socks://localhost:9050; open ftp://username:password@ftpserver"
Replace username
, password, and ftpserver with your FTP credentials and server details.
This command sets the ftp
:proxy
option to use Tor's SOCKS proxy on localhost
at port 9050
and then opens an FTP connection to the specified server using the provided credentials.
Note: If you have a specific Tor control port configured, you can also add the `--tor-ctrl-port` option to the command with the appropriate control port number.
Execute FTP commands: Once lftp connects through Tor, you can execute various FTP commands, such as ls
, get
, put
, etc
, to interact with the FTP server.
Tor Network
To use lftp with an Onion service (hidden service) on the Tor network, you need to configure lftp to connect through Tor's SOCKS proxy and specify the Onion service address. Here's how you can do it:
/etc/tor/torrc
Configure Tor SOCKS proxySocksPort 9050 SocksPolicy accept *
Start lftp with Tor: Use the following command to start lftp and configure it to use Tor's SOCKS proxy and connect to the Onion service:
user $
lftp -e "set ftp:proxy socks://localhost:9050; open ftp://username:password@onionaddress"
Replace username
, password
, and onionaddress with your FTP credentials and the Onion service address.
This command sets the ftp:proxy option to use Tor's SOCKS proxy on localhost at port 9050 and then opens an FTP connection to the specified Onion service using the provided credentials.
If you have a specific Tor control port configured, you can also add the
--tor-ctrl-port
option to the command with the appropriate control port number