User:Wuseman/lftp

From Gentoo Wiki
Jump to:navigation Jump to:search

LFTP is a sophisticated file transfer program supporting a number of network protocols (ftp, http, sftp, fish, torrent). Like BASH, it has job control and uses the readline library for input. It has bookmarks, a built-in mirror command, and can transfer several files in parallel. It was designed with reliability in mind. LFTP is free software, distributed under the GNU GPL license.

Install lftp

root #echo "gnutls ipv6 nls ssl convert-mozilla-cookies -idn socks5 -verify-file" >> tee /etc/portage/package.use/lftp
root #emerge --ask lftp

Basic Usage

Download all folders inside /foo/* on ftp to foo:
user $lftp -e open ftp://user1:pass1:site1:port -e "mirror -c -P1 remote/path /local/path" -d
FXP Between SITE 1 and SITE 2 with 20 threads at same time:{
user $lftp -e "set ftp:use-fxp true ;mirror -R -P20 ftp://user1:pass1:site1:port:/path ftp://user1:pass1:site1:port:/path" -d
Use lftp to multi-threaded download files from websites:
user $lftp -c "pget -n 10 http://example.com/foo.bar"
mirrors directory to a ftp server:
user $lftp -u 'login,passwd' -e "mirror reverse /my/from/dir/ /ftp/target/dir/" ftp://site.com/
Mirror a directory structure from websites with an Apache-generated file indexes:
user $lftp-e "mirror -c" http://example.com/foobar/
Internet Speed Test:
user $lftp -e 'pget http://address_to_file; exit;'
Multi-segment file downloading with lftp:
user $lftp -u "user,pass" ftp://site.com -e 'pget -c -n 6 file'
Multi-segment directory downloading with lftp:
user $lftp -u "user,pass" ftp://site.com/ -e 'mirror -c parallel=3 use-pget -n=5 "Some folder"'
Gets directory and files tree listing from a FTP-server:
user $lftp -u "user,pass" ftp://site.com/ -e "du -a;exit" > server-listing.txt
mirrors directory to a ftp server:
user $lftp -u login,passwd" -e "mirror reverse /my/from/dir/ /ftp/target/dir/" ftp://site.com/
Mirror a directory structure from websites with an Apache-generated file indexes:
user $lftp -e "mirror -c" http://example.com/foobar/
Internet Speed Test:
user $lftp-e 'pget http://address_to_file; exit;'
Multi-segment file downloading with lftp:
user $lftp -u "user,pass" ftp://site.com -e 'pget -c -n 6 file'
Multi-segment directory downloading with lftp:
user $lftp -u "user,pass" ftp://site.com/ -e 'mirror -c parallel=3 use-pget-n=5 "Some folder"'
Fastest segmented parallel sync of a remote directory over ssh
user $lftp -u "user,pass" -e "set sftp:connect-program 'ssh -a -x -T -c fooo -o Compression=no'; mirror -v -c loop use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22

Advanced Usage

┌┼───┤ lftp | ftp.sunet.se  ├───┤/├───
└┼─$─┤► : lftp -e open ftp://ftp.sunet.se -e "ls -sh about"
-rw-r--r--    1 ftp      ftp          2764 Sep 05  2022 ftp-about-SPmkII.html
-rw-r--r--    1 ftp      ftp          3297 Mar 29  2019 ftp.sunet.se-history_sv.html
-rw-r--r--    1 ftp      ftp          2550 May 07  2007 graph.png
-rw-r--r--    1 ftp      ftp          8629 Nov 12  2022 index.html
-rw-r--r--    1 ftp      ftp           279 May 07  2007 largefile.dot
-rw-r--r--    1 ftp      ftp          2550 May 07  2007 largefile.png
-rw-r--r--    1 ftp      ftp          4517 Mar 28  2019 largefile.svg
-rw-r--r--    1 ftp      ftp           210 May 07  2007 smallcachedfile.dot
-rw-r--r--    1 ftp      ftp          1558 May 07  2007 smallcachedfile.png
-rw-r--r--    1 ftp      ftp          3390 Mar 28  2019 smallcachedfile.svg
-rw-r--r--    1 ftp      ftp           303 May 07  2007 smalluncachedfile.dot
-rw-r--r--    1 ftp      ftp          2545 May 07  2007 smalluncachedfile.png
-rw-r--r--    1 ftp      ftp          4682 Mar 28  2019 smalluncachedfile.svg
drwxr-xr-x    2 ftp      ftp            31 Nov 14  2012 zfs-stats


We can check size as it was our own box and by adding quit to the command we can do things as it was our own shell, after the command has been executed then we disconnect so lets say we want to check size of a ftp dir 

lftp -e open "ftp://ftp.sunet.se" -e "du -sh about:quit"
3.6M    about

We can fxp files between fxps without downloading to our self, following command will send file from source_server to our destination_server

user $lftp -e "open -e 'fxp ftp://source_server.com; get /path/to/source/file' ftp://destination_server.com; quit"

We can paralllize a download for both download and fxp, here is a simple example for -c as in continue you can always continue if its broken

user $lftp -e open -u ${USER}:${PASSWORD} nr1.nu -p 65005 -e "mirror -c -P20 [/path/to/ftp_dir] [/path/to/local_dir]"

It is possible to build really advanced scripts for doing advanced multiple transfers between several sites in parallel, here is an example:

Example script for lftp
# Site 1
SITE1_HOST="site1.example.com"
SITE1_USER="username1"
SITE1_PASSWORD="password1"
SITE1_LOCAL_DIR="/path/to/local/dir"
SITE1_REMOTE_DIR="/path/to/remote/dir"

# Site 2
SITE2_HOST="site2.example.com"
SITE2_USER="username2"
SITE2_PASSWORD="password2"
SITE2_LOCAL_DIR="/path/to/local/dir"
SITE2_REMOTE_DIR="/path/to/remote/dir"

# Site 3
SITE3_HOST="site3.example.com"
SITE3_USER="username3"
SITE3_PASSWORD="password3"
SITE3_LOCAL_DIR="/path/to/local/dir"
SITE3_REMOTE_DIR="/path/to/remote/dir"

# Site 4
SITE4_HOST="site4.example.com"
SITE4_USER="username4"
SITE4_PASSWORD="password4"
SITE4_LOCAL_DIR="/path/to/local/dir"
SITE4_REMOTE_DIR="/path/to/remote/dir"

# FTP operations - Site 1
lftp -e "
open $SITE1_HOST
user $SITE1_USER $SITE1_PASSWORD
lcd $SITE1_LOCAL_DIR
cd $SITE1_REMOTE_DIR

# Upload files
put file1.txt
put file2.txt

# Download files
get file3.txt
get file4.txt

# Disconnect and quit
bye
"

# FTP operations - Site 2
lftp -e "
open $SITE2_HOST
user $SITE2_USER $SITE2_PASSWORD
lcd $SITE2_LOCAL_DIR
cd $SITE2_REMOTE_DIR

# Upload files
put file5.txt
put file6.txt

# Download files
get file7.txt
get file8.txt

# Disconnect and quit
bye
"

# FTP operations - Site 3
lftp -e "
open $SITE3_HOST
user $SITE3_USER $SITE3_PASSWORD
lcd $SITE3_LOCAL_DIR
cd $SITE3_REMOTE_DIR

# Upload files
put file9.txt
put file10.txt

# Download files
get file11.txt
get file12.txt

# Disconnect and quit
bye
"

# FTP operations - Site 4
lftp -e "
open $SITE4_HOST
user $SITE4_USER $SITE4_PASSWORD
lcd $SITE4_LOCAL_DIR
cd $SITE4_REMOTE_DIR

# Upload files
put file13.txt
put file14.txt

# Download files
get file15.txt
get file16.txt

# Disconnect and quit
bye
"

lftp supports SFTP, allowing you to transfer files in parallel without using SCP. 

set ftp:use-fxp true
set net:limit-total-rate 0:128000
set sftp:auto-confirm yes
set sftp:connect-program "ssh -a -x -T -c aes256-ctr -o Compression=no"

During the years I have been using tons of FTP clients for FXPing, for example, pftp and cftp and ftp with parallel and I tried most clients and I must say lftp is the master of the beast. However, it requires a lot of configuration to be properly set up for transferring files between 30-40-50 sites simultaneously. But once it's perfectly configured, I recommend it over anything else.

user $lftp sftp://username:password@host -e "mirror -R -P20 source target"

Well, there are tons of settings to use with lftp, so the best way to find them all is to read the manual: lftp Manual.

Anonymous Networking

To use lftp with Tor for anonymous and secure FTP connections, you can utilize Tor's SOCKS proxy feature. Here's how you can configure lftp to connect through Tor:

Configure Tor SOCKS proxy: Open the Tor configuration file (/etc/tor/torrc on Linux) and make sure the following lines are present:

FILE /etc/tor/torrcConfigure Tor SOCKS proxy
SocksPort 9050
SocksPolicy accept *


Start lftp with Tor: Use the following command to start lftp and configure it to use Tor's SOCKS proxy:

user $lftp -e "set ftp:proxy socks://localhost:9050; open ftp://username:password@ftpserver"

Replace username, password, and ftpserver with your FTP credentials and server details.

This command sets the ftp:proxy option to use Tor's SOCKS proxy on localhost at port 9050 and then opens an FTP connection to the specified server using the provided credentials.

Note: If you have a specific Tor control port configured, you can also add the `--tor-ctrl-port` option to the command with the appropriate control port number.

Execute FTP commands: Once lftp connects through Tor, you can execute various FTP commands, such as ls, get, put, etc, to interact with the FTP server.

Tor Network

To use lftp with an Onion service (hidden service) on the Tor network, you need to configure lftp to connect through Tor's SOCKS proxy and specify the Onion service address. Here's how you can do it:

FILE /etc/tor/torrcConfigure Tor SOCKS proxy
SocksPort 9050
SocksPolicy accept *

Start lftp with Tor: Use the following command to start lftp and configure it to use Tor's SOCKS proxy and connect to the Onion service:

user $lftp -e "set ftp:proxy socks://localhost:9050; open ftp://username:password@onionaddress"

Replace username, password, and onionaddress with your FTP credentials and the Onion service address.

This command sets the ftp:proxy option to use Tor's SOCKS proxy on localhost at port 9050 and then opens an FTP connection to the specified Onion service using the provided credentials.

Note
If you have a specific Tor control port configured, you can also add the --tor-ctrl-port option to the command with the appropriate control port number
Retrieved from "https://wiki.gentoo.org/index.php?title=User:Wuseman/lftp&oldid=1242459"