Google has added a new feature to the Google Cloud CLI to upload and download data to cloud storage buckets more efficiently and quickly. The Google article states (New gcloud storage CLI for your data transfers | Google Cloud Blog) that the new data transfers are up to 94% faster, comparing gsutil to gcloud storage. We did this test with rclone, gcloud storage and gsutil to see how fast gcloud storage really is.
Test environment:
For the tests, we used an n2d-standard-8 (8 vCPUs, 32 gb memory) VM and 375gb storage. Both the VM and the buckets were hosted in the us-east4 region (Northern Virginia).
For gsutil we used the –m flag which uses multiple threads. For rclone we also used the –transfers=N flag for transferring multiple files parallel.
Test case 1: Upload and download 100 files that are 100mb in size:
In the first test, 100 files were uploaded and downloaded. In this case the gcloud storage was the fastest in upload, rclone was fastest download, even faster than gcloud storage. The rclone was relatively constant in upload and download and the gsutil had a relatively fast upload but a very slow download.
Duration in Seconds for 100*100mb files, the lower the time the better
Test case 2: Upload and download 1 file that is 10gb in size:
In the second test, one file of 10gb was uploaded and downloaded. Again, the gcloud storage was the fastest. rclone was in the mid-field in both test cases and the gsutil was the slowest. The upload was significantly faster than the download in all cases, except rclone there upload was a lot slower than download. The difference between the gsutil and the gcloud storage was greatest for download.
Duration in Seconds for one 10gb file, the lower the time the better
Conclusion:
Summing up, we can say that the new gcloud storage CLI was the fastest in most of the tests. We were not able to reproduce the same difference of 94% between gsutil and gcloud storage. The rclone is placed in the mid-field, except when downloading many files. The rclone seems to be faster especially when used with many files, this may be because of the --transfers=N flag that runs multiple files parallel to each other.
