Fail to create buckets in Minio Object Storage due to missing region header #11073

Activity

1. abh1sar commented on Jun 26, 2025

   

   abh1sar

   on Jun 26, 2025

   Contributor

   More actions

   @levindecaro The MinIO Client used by cloudstack uses the default region us-east-1.
   Can you rename the region in your Minio cluster to 'us-east-1' and try?
2. abh1sar commented on Jun 26, 2025

   

   abh1sar

   on Jun 26, 2025

   Contributor

   More actions

   I guess it should work with the region ap-east-1 as well if the object store url is something like s3.ap-east-1.amazonaws.com so that minio client can infer the region from the url. Otherwise it will try to create the bucket in us-east-1.
3. levindecaro commented on Jun 26, 2025

   

   levindecaro

   on Jun 26, 2025

   ContributorAuthor

   More actions

   @abh1sar is it must start from s3 subdomain name? mine is apac-1a-s3.ap-east-1.example.com already
4. abh1sar commented on Jun 26, 2025

   

   abh1sar

   on Jun 26, 2025

   Contributor

   More actions

   yes, it needs to start with s3 and it should end with amazonaws or similar.
5. levindecaro commented on Jun 26, 2025

   

   levindecaro

   on Jun 26, 2025

   ContributorAuthor

   More actions

   @abh1sar I still got same failed result after adjusting fqdn, any suggestion?

   2025-06-26 23:27:53,882 DEBUG [o.a.c.s.o.BucketApiServiceImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Failed to create bucket with name: test-1 com.cloud.utils.exception.CloudRuntimeException: The authorization header is malformed; the region is wrong; expecting 'ap-east-1'.
           at org.apache.cloudstack.storage.datastore.driver.MinIOObjectStoreDriverImpl.createBucket(MinIOObjectStoreDriverImpl.java:125)
           at org.apache.cloudstack.storage.object.store.ObjectStoreImpl.createBucket(ObjectStoreImpl.java:115)
           at org.apache.cloudstack.storage.object.BucketApiServiceImpl.createBucket(BucketApiServiceImpl.java:147)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
           at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.base/java.lang.reflect.Method.invoke(Method.java:569)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
           at jdk.proxy3/jdk.proxy3.$Proxy442.createBucket(Unknown Source)
           at org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd.execute(CreateBucketCmd.java:188)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
           at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
           at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
           at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
           at java.base/java.lang.Thread.run(Thread.java:840)
   Caused by: error occurred
   ErrorResponse(code = AuthorizationHeaderMalformed, message = The authorization header is malformed; the region is wrong; expecting 'ap-east-1'., bucketName = test-1, objectName = null, resource = /test-1, requestId = 184CA145288407B8, hostId = 4fba84ceeb976175df772208f0c33f2701fa9e10940910b2ed75f362a95f469c)
   request={method=PUT, url=https://s3.ap-east-1.example.com/test-1, headers=Host: s3.ap-east-1.example.com
   Accept-Encoding: identity
   User-Agent: MinIO (Linux; amd64) minio-java/dev
   Content-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==
   x-amz-content-sha256: UNSIGNED-PAYLOAD
   x-amz-date: 20250626T152753Z
   Authorization: ██
   }
   response={code=400, headers=Accept-Ranges: bytes
   Content-Length: 409
   Content-Type: application/xml
   Server: MinIO
   Strict-Transport-Security: max-age=31536000; includeSubDomains
   Vary: Origin
   Vary: Accept-Encoding
   X-Amz-Bucket-Region: ap-east-1
   X-Amz-Id-2: 4fba84ceeb976175df772208f0c33f2701fa9e10940910b2ed75f362a95f469c
   X-Amz-Request-Id: 184CA145288407B8
   X-Content-Type-Options: nosniff
   X-Ratelimit-Limit: 1577
   X-Ratelimit-Remaining: 1577
   X-Xss-Protection: 1; mode=block
   Date: Thu, 26 Jun 2025 15:27:53 GMT
   }
   
           at io.minio.S3Base$1.onResponse(S3Base.java:690)
           at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
           ... 3 more
   
   2025-06-26 23:27:53,887 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Skipping delivering event Event {"description":"{\"details\":\"Bucket Id: 6181c9dd-15fc-40ac-aaa7-437074c3fb9c\",\"event\":\"BUCKET.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"BUCKET.CREATE","eventUuid":null,"resourceType":"Bucket","resourceUUID":null} to any webhook as account ID is missing
   2025-06-26 23:27:53,887 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906, ctx-b9b709d3]) (logid:30bdb6a7) Failed to publish event [category: ActionEvent, type: BUCKET.CREATE] on bus webhookEventBus
   2025-06-26 23:27:53,897 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Complete async job-137906, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create bucket with name: test-1. The authorization header is malformed; the region is wrong; expecting 'ap-east-1'."}
   2025-06-26 23:27:53,897 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Publish async job-137906 complete on message bus
   2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Wake up jobs related to job-137906
   2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Update db status for job-137906
   2025-06-26 23:27:53,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Wake up jobs joined with job-137906 and disjoin all subjobs created from job- 137906
   2025-06-26 23:27:53,903 DEBUG [c.c.a.ApiServer] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Retrieved cmdEventType from job info: BUCKET.CREATE
   2025-06-26 23:27:53,904 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Done executing org.apache.cloudstack.api.command.user.bucket.CreateBucketCmd for job-137906
   2025-06-26 23:27:53,905 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:[ctx-763f189b, job-137906]) (logid:30bdb6a7) Remove job-137906 from job monitoring
   
6. abh1sar commented on Jun 26, 2025

   

   abh1sar

   on Jun 26, 2025

   Contributor

   More actions

   @levindecaro which FQDN did you use?
7. levindecaro commented on Jun 26, 2025

   

   levindecaro

   on Jun 26, 2025

   Last edited by levindecaro

   ContributorAuthor

   More actions

   @levindecaro which FQDN did you use?

   s3.ap-east-1.[my company domain].com
8. abh1sar commented on Jun 27, 2025

   

   abh1sar

   on Jun 27, 2025

   Contributor

   More actions

   I looked at Minio sdk code.

           private boolean isAwsEndpoint(String endpoint) {
               return (endpoint.startsWith("s3.") || this.isAwsFipsEndpoint(endpoint) || this.isAwsAccelerateEndpoint(endpoint)) && (endpoint.endsWith(".amazonaws.com") || endpoint.endsWith(".amazonaws.com.cn"));
           }
           private void setBaseUrl(HttpUrl url) {
               String host = url.host();
               this.isAwsHost = this.isAwsEndpoint(host);
               this.isAwsChinaHost = false;
               if (this.isAwsHost) {
                   this.isAwsChinaHost = host.endsWith(".cn");
                   url = url.newBuilder().host(this.isAwsChinaHost ? "amazonaws.com.cn" : "amazonaws.com").build();
                   this.isFipsHost = this.isAwsFipsEndpoint(host);
                   this.isAccelerateHost = this.isAwsAccelerateEndpoint(host);
                   this.isDualStackHost = this.isAwsDualStackEndpoint(host);
                   this.regionInUrl = this.extractRegion(host);
                   this.useVirtualStyle = true;
               } else {
                   this.useVirtualStyle = host.endsWith("aliyuncs.com");
               }
   
               this.baseUrl = url;
           }
   

   it will refer the region from url only if isAwsEndpoint is true. There are some other logic in how they build the url which again takes amazonaws.com

   It might be difficult, but you can try if any strategy works by looking at the code.
   Otherwise using the region us-east-1 is a workaround.

   I don't think there is anything we can do from CLoudStack's side, as the logic is built into the Minio sdk.
9. levindecaro commented on Jun 27, 2025

   

   levindecaro

   on Jun 27, 2025

   ContributorAuthor

   More actions

   @abh1sar thank you for elaborate the underlying logic, let see what we can do behind that.
10. abh1sar commented on Jun 27, 2025

    

    abh1sar

    on Jun 27, 2025

    Contributor

    More actions

    @levindecaro thanks for acknowledging. I'll close this issue.
11. abh1sar

    closed this as completedon Jun 27, 2025
12. levindecaro commented on Jul 2, 2025

    

    levindecaro

    on Jul 2, 2025

    Last edited by levindecaro

    ContributorAuthor

    More actions

    @abh1sar I did some research on the Minio client. I believe it’s possible to add an optional region attribute to the MinioAdminClient by including the .region parameter during the connection establishment. What do you think?

    ./src/main/java/org/apache/cloudstack/storage/datastore/driver/MinIOObjectStoreDriverImpl.java

    https://github.com/minio/minio-java/blob/3835bee80ece65e2ed32db2027c8b74e346e3ef1/api/src/main/java/io/minio/MinioClient.java#L859
13. DaanHoogland

    added this to the unplanned milestone on Sep 20, 2025
14. abh1sar commented on Jun 24, 2026

    

    abh1sar

    on Jun 24, 2026

    Contributor

    More actions

    @abh1sar I did some research on the Minio client. I believe it’s possible to add an optional region attribute to the MinioAdminClient by including the .region parameter during the connection establishment. What do you think?

    ./src/main/java/org/apache/cloudstack/storage/datastore/driver/MinIOObjectStoreDriverImpl.java

    https://github.com/minio/minio-java/blob/3835bee80ece65e2ed32db2027c8b74e346e3ef1/api/src/main/java/io/minio/MinioClient.java#L859

    Hi @levindecaro , sorry I missed your note earlier. It does look like the bucket can be created using a specified region.
    Need to test if other functionalities will still work with that.
    We can probably let user specify a region while creating the object store.

    There is a feature request for this: #13326. We can track it there.