User:Wuseman/lftp

From Gentoo Wiki
Jump to:navigation Jump to:search

LFTP is a sophisticated file transfer program supporting a number of network protocols (ftp, http, sftp, fish, torrent). Like Bash, it has job control and uses the readline library for input. It has bookmarks, a built-in mirror command, and can transfer several files in parallel. It was designed with reliability in mind.

Installation

USE Flags

USE flags for net-ftp/lftp A sophisticated ftp/sftp/http/https/torrent client and file transfer program

+gnutls Prefer net-libs/gnutls as SSL/TLS provider (ineffective with USE=-ssl)
+ssl Add support for SSL/TLS connections (Secure Socket Layer / Transport Layer Security)
convert-mozilla-cookies Install the convert-mozilla-cookies script and its run-time dependencies
idn Enable support for Internationalized Domain Names
nls Add Native Language Support (using gettext - GNU locale utilities)
socks5 Add support for the socks5 proxy
verify-file Install the verify-file script and its run-time dependencies
Data provided by the Gentoo Package Database · Last update: 2025-02-23 22:16 More information about USE flags

Emerge

root #emerge --ask net-ftp/lftp

Usage

FXP with 20 Threads

Perform File eXchange Protocol (FXP) between two sites with 20 threads:

user $lftp -e "set ftp:use-fxp true ;mirror -R -P20 ftp://user1:pass1:site1:port:/path ftp://user1:pass1:site1:port:/path" -d

Multi-threaded Download

Download files from websites using multiple threads:

user $lftp -c "pget -n 10 http://example.com/foo.bar"

Directory Mirroring

Mirror a local directory structure to an FTP server:

user $lftp -u 'login,passwd' -e "mirror reverse /my/from/dir/ /ftp/target/dir/" ftp://site.com/

Apache Index Mirroring

Mirror a directory structure from websites with Apache-generated file indexes:

user $lftp-e "mirror -c" http://example.com/foobar/

Multi-segment File Download

Perform multi-segment file downloading:

user $lftp -u "user,pass" ftp://site.com -e 'pget -c -n 6 file'

Multi-segment Directory Download

Perform multi-segment directory downloading:

user $lftp -u "user,pass" ftp://site.com/ -e 'mirror -c parallel=3 use-pget -n=5 "Some folder"'

Directory and File Listing

Get a directory and file tree listing from an FTP server:

user $lftp -u "user,pass" ftp://site.com/ -e "du -a;exit" > server-listing.txt

Fast SSH Sync

Perform the fastest segmented parallel sync of a remote directory over SSH:

user $lftp -u "user,pass" -e "set sftp:connect-program 'ssh -a -x -T -c fooo -o Compression=no'; mirror -v -c loop use-pget-n=3 -P 2 /remote/dir/ /local/dir/; quit" sftp://remotehost:22

Advanced Usage

LFTP's capabilities extend beyond basic file transfers. It can be used to automate complex file transfer workflows across multiple servers. The following script demonstrates how to use LFTP to transfer files between multiple FTP servers. It allows you to define a source server from which files will be sent, along with multiple destination servers where the files will be transferred. The script initiates file transfers from the source server to each of the destination servers individually, ensuring that files are successfully delivered. Additionally, it enables file exchange between the destination servers themselves, allowing for

efficient and seamless distribution of files among all connected servers.

Here is a simple map for how it is done: 

  
     Source Server
         (source.example.com)
              |
              |   (Send file)
              v
     Destination Server 1
   (server1.example.com)
              |
              |   (Send file)
              v
     Destination Server 2
   (server2.example.com)
        /        \
  (Send file) (Send file)
      |             |
      v             v
Destination Server 3  Destination Server 4
(server3.example.com) (server4.example.com)
      |             |
      v             v
  (Send file)   (Send file)
      |             |
      v             v
Destination Server 5  Destination Server 6
(server5.example.com) (server6.example.com)

And here is an example script that implements this:

FILE /tmp/fileExample Script
#!/bin/bash
######################################################
# Script to Send Files Between Multiple FTP Servers  #
######################################################

source_server="source.example.com"
source_username="source_username"
source_password="source_password"
filename="file.txt"

destination_servers=("server1.example.com" "server2.example.com" "server3.example.com" "server4.example.com" "server5.example.com")
destination_username="destination_username"
destination_password="destination_password"
destination_directory="/path/to/destination/directory"

# Send the file from the source server to the five destination servers
for server in "${destination_servers[@]}"
do
    echo "Sending the file to $server"
    
    # Use lftp to connect to the source server and transfer the file to the destination server
    lftp -e "open -u $source_username,$source_password $source_server; put $filename -o $destination_directory/$filename; exit" $server &
done

wait

# Send files between the five destination servers
for ((i=0; i<${#destination_servers[@]}; i++)); do
    for ((j=i+1; j<${#destination_servers[@]}; j++))
    do
        echo "Sending the file from ${destination_servers[$i]} to ${destination_servers[$j]}"
          # Use lftp to connect to the first destination server, retrieve the file, and send it to the second destination server
        lftp -e "open -u $destination_username,$destination_password ${destination_servers[$i]}; get $filename -o - | lftp -e 'open -u $destination_username,$destination_password ${destination_servers[$j]}; put -O $destination_directory/$filename -; exit' -; exit" &
        echo "Sending the file from ${destination_servers[$j]} to ${destination_servers[$i]}"
        
        # Use lftp to connect to the second destination server, retrieve the file, and send it to the first destination server
        lftp -e "open -u $destination_username,$destination_password ${destination_servers[$j]}; get $filename -o - | lftp -e 'open -u $destination_username,$destination_password ${destination_servers[$i]}; put -O $destination_directory/$filename -; exit' -; exit" &
    done
done

wait

echo "All file transfers are completed"

LFTP allows us to interact with an FTP server as if it were our own local system. This gives us the ability to execute commands such as du -sh, which checks the size of a directory. By appending quit to the command, we ensure that LFTP exits after executing the command, thereby emulating a typical shell experience. Once the command has been executed, we automatically disconnect from the FTP server. For instance, to check the size of a directory on an FTP server:

user $lftp -e "open -e 'fxp ftp://source_server.com; get /path/to/source/file' ftp://destination_server.com; quit"

Configuration

user $cat ~/.lftprc

HTTP/HTTPS

While tools like curl and httpie are often the go-to choices for making HTTP/HTTPS POST and GET requests, it's worth noting that lftp can be equally capable and powerful for various tasks. Below is an example demonstrating how to perform a POST request to retrieve the "backup_jffs.tar" file from an Asus Router using lftp: 

lftp <<EOF
open http://192.168.50.1
quote set-cookie "asus_token=<session>t; clickedItem_tab=3"
set http:referer "http://192.168.50.1/Advanced_SettingBackup_Content.asp"
set http:use-compress true
set ssl:verify-certificate no
get /backup_jffs.tar -o backup_jffs.tar
EOF

In this script, lftp establishes a connection to "http://192.168.50.1", sets specific HTTP headers, and retrieves the "backup_jffs.tar" file, saving it as "backup_jffs.tar".

Tor SOCKS proxy

Open the Tor configuration file (/etc/tor/torrc on Linux) and make sure the following lines are present:

FILE /etc/tor/torrcConfigure Tor SOCKS proxy
SocksPort 9050
SocksPolicy accept *

Start lftp with Tor: Use the following command to start lftp and configure it to use Tor's SOCKS proxy:

user $lftp -e "set ftp:proxy socks://localhost:9050; open ftp://username:password@ftpserver"
Note
If there is a specific Tor control port configured, the user can incorporate the --tor-ctrl-port option into the command, providing the appropriate control port number.
Retrieved from "https://wiki.gentoo.org/index.php?title=User:Wuseman/lftp&oldid=1318729"