[GO] Handle file uploading with Rclone in a Golang project
Rclone is a great project that helps manage files across multi-cloud storage providers. This project has more than 32k stars on Github.
Rclone is a command-line program to manage files on cloud storage. It is a feature-rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
Rclone shipped as a command-line program and is easy for the end users to configure and run.
But what if you want to use the Rclone feature in your Golang project? Luckily, Rclone was written in Golang and it’s possible for us to directly use Rclone API in our project.
In this post, I will demonstrate the usage of the Rclone API to handle file uploading.
Assume that we already have an api that receives the Multipart form request, this api will take the request and expose a *multipart.FileHeader for us to use in the next step.
Because Rclone support lots of Storage Provider and we may want to use these providers in our code, we will need to create an abstract interface for the future implementation.
Define the type and interface
import (
"context"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/object"
)
type FileInfo struct {
Disk string `json:"disk,omitempty"`
Path string `json:"path,omitempty"`
Type string `json:"type,omitempty"`
Size int `json:"size,omitempty"`
}
type Disk interface {
Name() string
Url(filepath string) string
Delete(c context.Context, filepath string) error
Put(c context.Context, in io.Reader, size int64, mime, dst string) (*FileInfo, error)
Multipart(c context.Context, m *multipart.FileHeader, dsts ...string) (*FileInfo, error)
}
type RcloneDisk struct {
fs.Fs
DiskName string `json:"name"`
Root string
}
func (r *RcloneDisk) Name() string {
return r.DiskName
}
Diskis an interface that we will use laterFileInfoholds the uploaded file informationRcloneDiskis a base implementation of the Disk interface, the future disks we create will extend from this base
Take a look at Rclone Fs interface and we have this:
type Fs interface {
...
// Put in to the remote path with the modTime given of the given size
//
// When called from outside an Fs by rclone, src.Size() will always be >= 0.
// But for unknown-sized objects (indicated by src.Size() == -1), Put should either
// return an error or upload it properly (rather than e.g. calling panic).
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
Put(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
Rclone already implements the Fs interface for each supported provider, the full list can be found here
Create the Put method for the base struct RcloneDisk
func (r *RcloneDisk) Put(ctx context.Context, reader io.Reader, size int64, mime, dst string) (*FileInfo, error) {
objectInfo := object.NewStaticObjectInfo(
dst,
time.Now(),
size,
true,
nil,
nil,
)
rs, err := r.Fs.Put(ctx, reader, objectInfo)
if err != nil {
return nil, err
}
return &FileInfo{
Disk: r.DiskName,
Path: dst,
Type: mime,
Size: int(rs.Size()),
}, nil
}
As defined in the Rclone Fs interface, the Put method receives these parameters:
ctx: A context for easily controlling the method behaviorreader: Anio.Readerobject that holds the file upload reader, we will get this reader from the*multipart.FileHeadersrc: Arclonefs.ObjectInfoobject that holds the file information. In order to create this object will have to collect some additional information as file size, upload destination
Create the Multipart method for the base struct RcloneDisk
func (r *RcloneDisk) Multipart(ctx context.Context, m *multipart.FileHeader, dsts ...string) (*FileInfo, error) {
f, err := m.Open()
if err != nil {
return nil, err
}
fileHeader := make([]byte, 512)
if _, err := f.Read(fileHeader); err != nil {
return nil, err
}
if _, err := f.Seek(0, 0); err != nil {
return nil, err
}
dst := ""
mime := http.DetectContentType(fileHeader)
if len(dsts) > 0 {
dst = dsts[0]
} else {
dst = r.UploadFilePath(m.Filename)
}
return r.Put(ctx, f, m.Size, mime, dst)
}
Create the default file path for every upload
func (r *RcloneDisk) UploadFilePath(filename string) string {
var filenameRemoveCharsRegexp = regexp.MustCompile(`[^a-zA-Z0-9_\-\.]`)
var dashRegexp = regexp.MustCompile(`\-+`)
now := time.Now()
filename = filenameRemoveCharsRegexp.ReplaceAllString(filename, "-")
filename = dashRegexp.ReplaceAllString(filename, "-")
filename = strings.ReplaceAll(filename, "-.", ".")
return path.Join(
r.Root,
strconv.Itoa(now.Year()),
fmt.Sprintf("%02d", int(now.Month())),
fmt.Sprintf("%d_%s", now.UnixMicro(), filename),
)
}
func (r *RcloneDisk) Delete(ctx context.Context, filepath string) error {
obj, err := r.Fs.NewObject(ctx, filepath)
if err != nil {
return err
}
return obj.Remove(ctx)
}
Note:
Because will Read the file to check its mime f.Read(fileHeader), we have to seek it at the start of the io.Reader to avoid file corruption: f.Seek(0, 0)
The basic implementation of the RcloneDisk is done, we will create the S3 Disk and the Local Disk that extend this base disk in the steps bellow.
Create the local disk: Save file to the local storage
type RcloneLocal struct {
*RcloneDisk
Root string `json:"root"`
BaseUrl string `json:"base_url"`
}
func (r *RcloneLocal) Url(filepath string) string {
return r.BaseUrl + "/" + filepath
}
This struct contains extra properties:
Root: The root directory for the uploadBaseurl: The base url for the public file url
Create the local disk:
import (
"context"
"os"
"github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/fs/config/configmap"
)
func NewLocal() Disk {
rl := &RcloneLocal{
BaseRcloneDisk: &BaseRcloneDisk{
DiskName: "local_disk",
},
Root: "/var/www/html/files",
BaseUrl: "/files",
}
cfgMap := configmap.New()
cfgMap.Set("root", rl.Root)
fsDriver, err := local.NewFs(context.Background(), rl.DiskName, rl.Root, cfgMap)
if err != nil {
panic(err)
}
rl.Fs = fsDriver
return rl
}
In this code, we import the Local backend that implemented the Rclone Fs: "github.com/rclone/rclone/backend/local"
Create the S3 disk: Save file to AWS S3 or S3 compatible storage providers
type RcloneS3 struct {
*RcloneDisk
Root string `json:"root"`
Provider string `json:"provider"`
Bucket string `json:"bucket"`
Region string `json:"region"`
Endpoint string `json:"endpoint"`
ChunkSize fs.SizeSuffix `json:"chunk_size"`
AccessKeyID string `json:"access_key_id"`
SecretAccessKey string `json:"secret_access_key"`
BaseUrl string `json:"base_url"`
ACL string `json:"acl"`
}
func (r *RcloneS3) Url(filepath string) string {
return r.BaseUrl + filepath
}
Create the S3 disk
import (
"context"
"github.com/rclone/rclone/backend/s3"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
)
func NewS3(cfg *RcloneS3Config) fs.FSDisk {
rs3 := &RcloneS3{
BaseRcloneDisk: &BaseRcloneDisk{
DiskName: "DO_DISK",
Root: "/files",
},
Root: "/files",
Provider: "DigitalOcean",
Region: "sfo3",
Endpoint: "sfo3.digitaloceanspaces.com",
ChunkSize: rclonefs.SizeSuffix(1024 * 1024 * 5),
AccessKeyID: "AccessKeyID",
SecretAccessKey: "SecretAccessKey",
BaseUrl: "https://cdn.mysite",
ACL: "public-read",
}
cfgMap := &configmap.Simple{}
cfgMap.Set("provider", rs3.Provider)
cfgMap.Set("bucket", rs3.Bucket)
cfgMap.Set("region", rs3.Region)
cfgMap.Set("endpoint", rs3.Endpoint)
cfgMap.Set("chunk_size", rs3.ChunkSize.String())
cfgMap.Set("access_key_id", rs3.AccessKeyID)
cfgMap.Set("secret_access_key", rs3.SecretAccessKey)
cfgMap.Set("acl", rs3.ACL)
cfgMap.Set("bucket_acl", rs3.ACL)
fsDriver, err := s3.NewFs(context.Background(), "s3", rs3.Bucket, cfgMap)
if err != nil {
panic(err)
}
rs3.Fs = fsDriver
return rs3
}
In this code, we import the S3 backend that implemented the Rclone Fs: "github.com/rclone/rclone/backend/s3"
Usage
var localDisk = NewLocal()
var s3Disk = NewS3()
func Upload(c server.Context) error {
ctx := context.Background()
fileHeader := c.File("file") // *multipart.FileHeader
localDisk.Multipart(ctx, fileHeader) // Save file to the local storage
s3Disk.Multipart(ctx, fileHeader) // Save file to the S3 compatible storage
}
Conclusion
In this post, we just talked about the File Saving process using Rclone, there are many features that Rclone offers. By looking into the source code we can directly use Rclone APIs without using its binary.
Thank you for spending time on my post, if you have any questions, don’t hesitate to leave a comment here!
ádasd