# Wednesday, December 2, 2020

Recently, I have been working with MinIO – a container-based object storage service. I’ve recorded much of what I’ve learned. The links below will get you up to speed if you want to learn about this technology.

Blog Posts

Getting Started with MinIO Server

Creating and using a MinIO Gateway for Azure

Using the MinIO Java SDK

Managing MinIO with the Amazon S3 SDK

Videos

Creating a MinIO Server

Creating a MinIO Agent for Azure Blob Storage

Using the MinIO Java Client SDK

Accessing MinIO with the AWS S3 SDK

Wednesday, December 2, 2020 9:15:00 AM (GMT Standard Time, UTC+00:00)
# Tuesday, December 1, 2020

Intro

In my last article, I showed how to manage buckets and objects in MinIO using the MinIO Java SDK.

However, MinIO has the advantage that one can also access it using the Amazon S3 Java API. This is helpful if you are migrating from S3 (a comparable object store hosted by Amazon Web Services) to MinIO.

The code below assumes that the following values are declared and initialized appropriately:

private String endPoint;  // The MinIO endpoint (e.g., "http://127.0.0.1:9000") 
private String accessKey; // The MinIO Access Key 
private String secretKey; // The MinIO Secret Key 
private String bucketName; // A MinIO bucket in which to store objects (e.g., "mybucket") 
private String localFileFolder; // A local folder on your file system to upload/download files to/from MinIO (e.g., "c:\files\")
  

In order to use the S3 SDK, your app must have a reference to it. In a Maven project, this is done by adding the following to the <dependencies> section of the project's POM.XML:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.858</version>
</dependency>
  

AmazonS3 Object

In your code, the first thing you will need is an AmazonS3 object, which has methods for managing your MinIO objects.

Here is the code for creating this object.

public static AmazonS3 getAmazonS3Client(String accessKey, String secretKey, String endPoint) { 
    ClientConfiguration clientConfig = new ClientConfiguration(); 
    clientConfig.setProtocol(Protocol.HTTP); 
    AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); 
    AmazonS3 s3client = AmazonS3ClientBuilder 
            .standard() 
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endPoint, Regions.US_EAST_1.name())) 
            .withPathStyleAccessEnabled(true) 
            .withClientConfiguration(clientConfig) 
             .withCredentials(new AWSStaticCredentialsProvider(credentials)) 
            .build();

    return s3client; 
}
  

Once you have an AmazonS3 object, you can use it to manage MinIO objects.

Uploading a File

For example, here is code to upload a file to a MinIO bucket:

public void UploadWithS3Client(String fileName) throws IOException { 
     AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint); 
    String fileToUpload = localFileFolder + fileName; 
     try { 
        File file = new File(fileToUpload); 
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, fileName, file); 
        s3Client.putObject(putObjectRequest); 
    } catch (AmazonServiceException ase) { 
        System.out.println("Error Message:    " + ase.getMessage());

    } catch (AmazonClientException ace) { 
        System.out.println("Error Message: " + ace.getMessage()); 
    } 
}
  

List Objects

and code to get a list all the objects in a bucket

public List<String> ListS3Objects() { 
     List<String> blobList = new ArrayList<String>(); 
    System.out.format("Objects in S3 bucket %s:\n", bucketName); 
    AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint); 
     ListObjectsV2Result result = s3Client.listObjectsV2(bucketName); 
    List<S3ObjectSummary> blobs = result.getObjectSummaries(); 
    for (S3ObjectSummary blob : blobs) { 
        blobList.add(blob.getKey()); 
        System.out.println("* " + blob.getKey()); 
    } 
    return blobList; 
 }
  

Download a File

and to download one of those objects to your local file system:

public void DownloadFromMinIOWithS3Client(String objectName) { 
    System.out.format("Downloading %s from S3 bucket %s...\n", objectName, bucketName); 
    AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint); 
    try { 
        S3Object o = s3Client.getObject(bucketName, objectName); 
        S3ObjectInputStream s3is = o.getObjectContent(); 
        String downloadedFile = localFileFolder + "D_" + objectName; 
        FileOutputStream fos = new FileOutputStream(new File(downloadedFile)); 
        byte[] read_buf = new byte[1024]; 
        int read_len = 0; 
        while ((read_len = s3is.read(read_buf)) > 0) { 
            fos.write(read_buf, 0, read_len); 
        } 
        s3is.close(); 
        fos.close(); 
    } catch (AmazonServiceException e) { 
        System.err.println(e.getErrorMessage()); 
        System.exit(1); 
    } catch (FileNotFoundException e) { 
        System.err.println(e.getMessage()); 
         System.exit(1); 
    } catch (IOException e) { 
        System.err.println(e.getMessage()); 
        System.exit(1); 
        } 
    }
  

As you can see, once you have a reference to the object, the rest is just Java IO code.

Print File Contents

Finally, here is code to print the contents of a text object stored in MinIO. Again, it is simple Java IO once you have a reference to the object.

    public void PrintObjectContents(String objectName) throws IOException {
        AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint);
        GetObjectRequest rangeObjectRequest = new GetObjectRequest(bucketName, objectName);
        S3Object objectPortion = s3Client.getObject(rangeObjectRequest);
        System.out.println("Printing bytes retrieved:");
        displayTextInputStream(objectPortion.getObjectContent());
    }

    private static void displayTextInputStream(InputStream input) throws IOException {
        BufferedReader reader = new BufferedReader(new InputStreamReader(input));
        while (true) {
            String line = reader.readLine();
            if (line == null)
                break;

            System.out.println("    " + line);
        }
        System.out.println();
    }
  

Conclusion

Here is the full code that you can find at https://github.com/DavidGiard/MinIO_Java_Demo:

package com.gcast.gcastminio.services;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.ListObjectsV2Result;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3ObjectSummary;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class S3Service {

    // The following are set in application.properties
    @Value("${minio.endPoint}")
    private String endPoint;
    @Value("${minio.accessKey}")
    private String accessKey;
    @Value("${minio.secretKey}")
    private String secretKey;
    @Value("${minio.bucketName}")
    private String bucketName;
    @Value("${localFileFolder}")
    private String localFileFolder;

    public void UploadWithS3Client(String fileName) throws IOException {
        AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint);
        String fileToUpload = localFileFolder + fileName;
        try {
            File file = new File(fileToUpload);

            PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, fileName, file);
            s3Client.putObject(putObjectRequest);
        } catch (AmazonServiceException ase) {
            System.out.println("Error Message:    " + ase.getMessage());

        } catch (AmazonClientException ace) {
            System.out.println("Error Message: " + ace.getMessage());
        }
    }

    public List<String> ListS3Objects() {
        List<String> blobList = new ArrayList<String>();
        System.out.format("Objects in S3 bucket %s:\n", bucketName);
        AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint);
        ListObjectsV2Result result = s3Client.listObjectsV2(bucketName);
        List<S3ObjectSummary> blobs = result.getObjectSummaries();
        for (S3ObjectSummary blob : blobs) {
            blobList.add(blob.getKey());
            System.out.println("* " + blob.getKey());
        }
        return blobList;
    }

    public void PrintObjectContents(String objectName) throws IOException {
        AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint);
        GetObjectRequest rangeObjectRequest = new GetObjectRequest(bucketName, objectName);
        S3Object objectPortion = s3Client.getObject(rangeObjectRequest);
        System.out.println("Printing bytes retrieved:");
        displayTextInputStream(objectPortion.getObjectContent());
    }

    private static void displayTextInputStream(InputStream input) throws IOException {
        BufferedReader reader = new BufferedReader(new InputStreamReader(input));
        while (true) {
            String line = reader.readLine();
            if (line == null)
                break;

            System.out.println("    " + line);
        }
        System.out.println();
    }

	public void DownloadFromMinIOWithS3Client(String objectName) {
		System.out.format("Downloading %s from S3 bucket %s...\n", objectName, bucketName);
		AmazonS3 s3Client = getAmazonS3Client(accessKey, secretKey, endPoint);
		try {
			S3Object o = s3Client.getObject(bucketName, objectName);
			S3ObjectInputStream s3is = o.getObjectContent();
			String downloadedFile = localFileFolder + "D_" + objectName;
			FileOutputStream fos = new FileOutputStream(new File(downloadedFile));
			byte[] read_buf = new byte[1024];
			int read_len = 0;
			while ((read_len = s3is.read(read_buf)) > 0) {
				fos.write(read_buf, 0, read_len);
			}
			s3is.close();
			fos.close();
		} catch (AmazonServiceException e) {
			System.err.println(e.getErrorMessage());
			System.exit(1);
		} catch (FileNotFoundException e) {
			System.err.println(e.getMessage());
			System.exit(1);
		} catch (IOException e) {
			System.err.println(e.getMessage());
			System.exit(1);
			}
		}


    public static AmazonS3 getAmazonS3Client(String accessKey, String secretKey, String endPoint) {
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setProtocol(Protocol.HTTP);
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
        AmazonS3 s3client = AmazonS3ClientBuilder
                .standard()
                .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endPoint, Regions.US_EAST_1.name()))
                .withPathStyleAccessEnabled(true)
                .withClientConfiguration(clientConfig)
                .withCredentials(new AWSStaticCredentialsProvider(credentials))
                .build();

        return s3client;
	}
}
  

In this article, you learned how to use the Amazon S3 Java SDK to manage objects in MinIO.

Tuesday, December 1, 2020 7:23:00 AM (GMT Standard Time, UTC+00:00)
# Wednesday, November 25, 2020

Intro

In previous articles, I showed how to configure a MinIO Server and a MinIO Gateway for Azure.

In those articles, I demonstrated how to manage objects using the MinIO user interface.

In this article, I will show how to use the MinIO Java API to manage MinIO objects.

In order to use the MinIO API, you will need to set a dependency in your project. In a Maven project, this is done by adding the following to the <dependencies> section of the project's POM.XML:

<dependency>
  <groupId>io.minio</groupId>
  <artifactId>minio</artifactId>
  <version>7.1.2</version>
</dependency>
  

The code below assumes that the following values are declared and initialized appropriately:

private String endPoint;        // The MinIO endpoint (e.g., "http://127.0.0.1:9000")
private String accessKey;       // The MinIO Access Key
private String secretKey;       // The MinIO Secret Key
private String bucketName;    // A MinIO bucket in which to store objects (e.g., "mybucket")
private String localFileFolder; // A local folder on your file system to upload/download files to/from MinIO (e.g., "c:\files\")

You will also need to import namespaces from the subnamespaces of min.io. Your IDE will help you identify variables that require these namespaces.   

The MinIOClient object

In your code, you will need a MinioClient object to work with MinIO. A builder helps with this.

MinioClient minioClient = MinioClient.builder().endpoint(endPoint).credentials(accessKey, secretKey).build();
  

Buckets

Below is the code to check if a bucket already exists and create it if it does not

boolean bucketExists = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build()); 
if (!bucketExists) { 
    minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build()); 
}
  

Notice the arguments passed to bucketExists. These arguments are creased with a builder (BucketExistsArgs.builder().bucket(bucketName).build()). The arguments for makeBucket use a similar pattern. We will see this pattern in a lot of MinioClient methods.

Uploading a File

Here is the code to upload a file into a MinIO bucket.

String fileName = "file1.txt"; // This file must exist in the local file folder 
String fileToUpload = localFileFolder + fileName; 
UploadObjectArgs args = UploadObjectArgs.builder().bucket(bucketName).object(fileName).filename(fileToUpload).build(); 
minioClient.uploadObject(args);
  

Downloading a File

Here is the code to download an object to a file on your local drive:

String fileName = "file1.txt"; // This file must exist in the MinIO bucket 
String downloadedFile = localFileFolder + "D_" + fileName; 
DownloadObjectArgs args = DownloadObjectArgs.builder().bucket(bucketName).object(fileName) 
        .filename(downloadedFile).build(); 
minioClient.downloadObject(args);
  

Conclusion

The full listing of a class to read and write to MinIO is shown in Listing 1

Listing 1:

package com.gcast.gcastminio.services;

import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import io.minio.BucketExistsArgs;
import io.minio.DownloadObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.UploadObjectArgs;
import io.minio.errors.MinioException;

@Service
public class MinIOService {

    // The following are set in application.properties
    @Value("${minio.endPoint}")
    private String endPoint;
    @Value("${minio.accessKey}")
    private String accessKey;
    @Value("${minio.secretKey}")
    private String secretKey;
    @Value("${minio.bucketName}")
    private String bucketName;
    @Value("${localFileFolder}")
    private String localFileFolder;

    public void WriteToMinIO(String fileName)
            throws InvalidKeyException, IllegalArgumentException, NoSuchAlgorithmException, IOException {
        try {
            MinioClient minioClient = MinioClient.builder().endpoint(endPoint)
                    .credentials(accessKey, secretKey).build();

            boolean bucketExists = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
            if (!bucketExists) {
                minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
            }

            String fileToUpload = localFileFolder + fileName;
            UploadObjectArgs args = UploadObjectArgs.builder().bucket(bucketName).object(fileName)
                    .filename(fileToUpload).build();
            minioClient.uploadObject(args);

            System.out.println(fileToUpload + " successfully uploaded to:");
            System.out.println("   container: " + bucketName);
            System.out.println("   blob: " + fileName);
            System.out.println();
        } catch (MinioException e) {
            System.out.println("Error occurred: " + e);
        }
    }

    public void ReadFromMinIO(String fileName)
            throws InvalidKeyException, IllegalArgumentException, NoSuchAlgorithmException, IOException {
        try {
            MinioClient minioClient = MinioClient.builder().endpoint(endPoint)
                    .credentials(accessKey, secretKey).build();
            String downloadedFile = localFileFolder + "D_" + fileName;
            DownloadObjectArgs args = DownloadObjectArgs.builder().bucket(bucketName).object(fileName)
                    .filename(downloadedFile).build();
            minioClient.downloadObject(args);

            System.out.println("Downloaded file to ");
            System.out.println(" " + downloadedFile);
            System.out.println();
        } catch (MinioException e) {
            System.out.println("Error occurred: " + e);
        }
    }
    
}
  

You can download and run  this code from https://github.com/DavidGiard/MinIO_Java_Demo

The code is the same whether you are running a MinIO Server or MinIO Gateway.

In this article, you learned how to use the MinIO Java SDK to read and write objects stored in MinIO.

Database | Java | MinIO
Wednesday, November 25, 2020 9:56:00 AM (GMT Standard Time, UTC+00:00)
# Tuesday, November 24, 2020

What is MinIO?


MinIO is an object storage system, similar to Amazon S3 or Azure Blob storage. It is built on top of Docker containers, which makes it easy to scale.

In a previous article, I showed you how to create and use a MinIO Server.

In this article, I will show how to create and use a MinIO Gateway for Azure Blob Storage.

MinIO Gateway

A MinIO Server stores files and objects. By contrast, a MinIO points to some other storage repository where the files are stored. However, it allows you to interact with those files as if they were stored in MinIO.

Prerequisites

Before you begin, you will need to install Docker Desktop, which you can download for either Windows or Mac.

You will also need an Azure Storage Account. This article explains how to create an Azure Storage Account.

Azure Blob Storage

You will need two pieces of information from your Azure Storage Account: the name of the storage account and the access key.

In the Azure Portal (https://portal.azure.com), you can find the storage account name at the top of the Resource page, as shown in Fig. 1.

mga01-StorageAccountName
Fig. 1

You can find the key on the "Access Keys" blade, as shown in Fig. 2.

mga02-StorageAccountKeys
Fig. 2

Note that there are two keys. Either one will work. Click the [Show Keys] button to view the keys and allow copying to your clipboard.

Creating a MinIO Gateway

A MinIO Gateway for Azure is created with the following command:

docker run -p 9000:9000 --name azure-s3 -e "MINIO_ACCESS_KEY=azurestorageaccountname" -e "MINIO_SECRET_KEY=azurestorageaccountkey" minio/minio gateway azure

where

azurestorageaccountname is the name of the Azure storage account and azurestorageaccountkey is an account key from that same storage account.

You can now log into the MinIO Gateway by opening a browser and navigating to http://127.0.0.1:9000/.

When prompted for your login credentials (Fig. 3), enter the storage account name in the "Access key" field and enter the storage account key in the "Secret Key" field.

mga03-Login
Fig. 3

After a successful login, the MinIO Gateway user interface will display, as shown in Fig. 4.

mga04-MinIO

Fig. 4

Note that this looks exactly like the MinIO Server user interface, described in this article.

In fact, you can create buckets and manage files in a MinIO Gateway exactly as you would in a MinIO server. The only difference is that the objects you manipulate are stored in the corresponding Azure Blob storage, rather than in MinIO. Each bucket is mapped to a Blob Storage container and each file is mapped to a blob.

Conclusion

In this article, you learned how to create a MinIO Gateway for Azure.

Tuesday, November 24, 2020 9:31:00 AM (GMT Standard Time, UTC+00:00)
# Thursday, November 19, 2020

What is MinIO?

MinIO is an object storage system, similar to Amazon S3 or Azure Blob storage. It is built on top of Docker containers, which makes it easy to scale.

Because MinIO runs in a Docker container, it requires the installation of Docker.

You can either install the Docker engine here or install Docker Desktop at one of the following links:

Starting a MinIO Server

Once Docker is installed, use the following command to start a MinIO server:

docker run -p 9000:9000 -e "MINIO_ACCESS_KEY=myAccessKey" -e "MINIO_SECRET_KEY=mySecretKey" minio/minio server /data

You can replace myAccessKey and mySecretKey with just about any string you like. These will be used to log into the MinIO server. Write down these values and keep them in a safe place! You will need them in order to access your server.

After you run the above command, you can access the server's UI by opening a web browser and navigating to

http://127.0.0.1:9000/

(NOTE: Of course, you may choose to run your server on a different port than 9000. If so, modify the “Docker” command above.)

You will be prompted to log in, as shown in Fig. 1

mio01Login
Fig. 1

Enter the access key and secret key you selected in the Docker command above.

After successfully logging in, you will see the MinIO user interface, as shown in Fig. 2.

mio02-MinIOServer
Fig. 2

MinIO organizes objects into buckets, which are analogous to folders in a file system or containers in Azure blob storage. To create a new bucket, click the [+] icon (Fig. 3) in the lower right of the screen.

mio03-PlusButton
Fig. 3

A popup menu will display, as shown in Fig. 4.

mio04-Menu
Fig. 4

Click the [Create Bucket] icon (Fig. 5) to display the "New Bucket" dialog, as shown in Fig. 6.

mio05-CreateBucketIcon
Fig. 5

mio06-BucketName
Fig. 6

In the "New Bucket" dialog, enter a name for your bucket, as shown in Fig. 7. This name must be unique within this MinIO server, must be at least 3 characters long and may consist only of numbers, periods, hyphens, and lower-case letters. 

mio07-BucketName
Fig. 7

Press ENTER to create the bucket. The Bucket will now be listed along the left side, as shown in Fig. 8.

mio08-Bucket
Fig. 8

You can add files to this bucket by again clicking the lower-left [+] icon to display the popup menu shown in Fig. 9.

mio09-Menu
Fig. 9

Click the "Upload File" icon (Fig. 10) to open a File Selection dialog, as shown in Fig. 11.

mio10-UploadFile
Fig. 10

mio11-SelectFile
Fig. 11

Navigate to and select a file on your local drive and click the [Open] button. The file will be listed within the bucket, as shown in Fig. 12.

mio12-ListFiles
Fig. 12

You can click the […] at the right of the file listing row to expand a menu with options to share, preview, download, or delete the file from MinIO, as shown in Fig. 13

mio13-Menu
Fig. 13

In this article, I introduced MinIO Server and showed the basics of getting started and using this object storage tool.

Thursday, November 19, 2020 9:02:00 AM (GMT Standard Time, UTC+00:00)
# Thursday, October 29, 2020

GCast 97:

Accessing MinIO with the AWS S3 SDK

The AWS S3 SDK for Java allows you to read from and write to MinIO. This allows you to easily migrate an application from using S3 to using MinIO Server or Agent.

Code:

https://github.com/DavidGiard/MinIO_Java_Demo/releases/tag/GCast097

Database | GCast | Java | MinIO | Screencast | Video
Thursday, October 29, 2020 8:24:00 AM (GMT Standard Time, UTC+00:00)
# Thursday, October 1, 2020

GCast 96:

Using the MinIO Java Client SDK

Learn how to use the Java Client SDK to upload and download files to/from a MinIO server

Code: https://github.com/DavidGiard/MinIO_Java_Demo/releases/tag/GCast096

Database | GCast | Java | MinIO | Screencast | Video
Thursday, October 1, 2020 9:49:00 AM (GMT Daylight Time, UTC+01:00)
# Thursday, September 24, 2020

GCast 95:

Creating a MinIO Agent for Azure Blob Storage

Learn how to use MinIO to manage blobs in an Azure Storage Account

Thursday, September 24, 2020 12:25:40 PM (GMT Daylight Time, UTC+01:00)
# Thursday, September 17, 2020

GCast 94:

Creating a MinIO Server

Learn how to create a MinIO server, organize into buckets; then, read and write files to the server.

Thursday, September 17, 2020 9:21:00 AM (GMT Daylight Time, UTC+01:00)