Skip to content

Default systemVM template is not downloaded to S3 secondary storage #9002

New issue
New issue
Open
Bug
#12426
Open
Default systemVM template is not downloaded to S3 secondary storage#9002
Bug
#12426
Assignees
Damans227
Labels
component:secondary-storagecomponent:systemvm-templateno-issue-activity
Milestone
4.20.4

Description

@rajujith
rajujith
opened on Apr 29, 2024
ISSUE TYPE
  • Bug Report
COMPONENT NAME
S3 Secondary storage, system VM template
CLOUDSTACK VERSION
4.19 with PR #8986
CONFIGURATION

MinIO S3 bucket configured
NFS staging storage

SUMMARY

Although the S3 secondary storage gets added with the fix in #8986 the default system VM template do not get downloaded. In the management server logs, I can see the 'DownloadProgressCommand' running but the file is not being downloaded. Note that the zone is created fresh and there is no SSVM to download the template. I am assuming there may be something missing to copy the systemVM template in the management server to the S3 bucket either via staging NFS or without it.

STEPS TO REPRODUCE
1. Create a MinIO S3 bucket
2. Add a zone with S3 secondary storage and NFS staging storage using the above bucket.
3. Launch the zone and wait for the system VM template status to become 'Ready'
EXPECTED RESULTS
SystemVM template is downloaded
ACTUAL RESULTS
SystemVM template is not downloaded

Activity

  1. rajujith
    added
    component:secondary-storage
    component:systemvm-template
    on Apr 29, 2024
  2. yadvr
    added this to the 4.19.1.0 milestone on Apr 30, 2024

  3. yadvr commented on Apr 30, 2024

    @yadvr
    yadvr
    on Apr 30, 2024
    Member

    Worth checking if that's supported at all.

  4. DaanHoogland commented on May 2, 2024

    @DaanHoogland
    DaanHoogland
    on May 2, 2024
    Contributor

    @swill @pdion891 can you say whether this is supposed to work? and how?

  5. sureshanaparti
    added this to Apache CloudStack 4.19.1on May 23, 2024
  6. sureshanaparti
    moved this to Todo in Apache CloudStack 4.19.1on May 23, 2024
  7. sureshanaparti
    added this to Apache CloudStack BugFest - Issueson Jun 3, 2024
  8. sureshanaparti
    moved this to Todo in Apache CloudStack BugFest - Issueson Jun 3, 2024
  9. sureshanaparti
    modified the milestones: 4.19.1.0, 4.19.2.0 on Jun 25, 2024
  10. sureshanaparti
    removed this from Apache CloudStack 4.19.1on Jun 25, 2024
  11. sureshanaparti
    added this to deprecated: Apache CloudStack - Issues Aug'2024on Aug 9, 2024
  12. DaanHoogland
    moved this to Todo in deprecated: Apache CloudStack - Issues Aug'2024on Aug 13, 2024
  13. DaanHoogland
    moved this from Todo to No status in deprecated: Apache CloudStack - Issues Aug'2024on Aug 14, 2024

  14. 7 remaining items

  15. weizhouapache
    added this to the 4.20.2 milestone on Sep 2, 2025
  16. weizhouapache
    modified the milestones: 4.20.2, 4.20.3 on Oct 23, 2025
  17. DaanHoogland
    added the
    Bug
    issue type on Jan 6, 2026
  18. nvazquez
    moved this from Todo to Discuss in Apache CloudStack BugFest - Issueson Jan 12, 2026

  19. Damans227 commented on Jan 13, 2026

    @Damans227
    Damans227
    on Jan 13, 2026
    Collaborator

    I dug into this a bit. I set up a fresh 4.20.3 env with MinIO running on one of the management server VMs and configured an S3 bucket for secondary storage.

    Image

    The secondary storage gets created successfully, but the system VM never comes up.

    In the logs I’m seeing this error: System VM Template is not ready at zone [1], wait until it is ready to launch secondary storage VM.

    2026-01-13 17:47:53,998 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-49a74801]) (logid:6b471c7c) System VM Template is not ready at zone [1], wait until it is ready to launch secondary storage VM.
2026-01-13 17:48:23,816 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-839ca70c]) (logid:41b5066c) System vm template is not ready at data center Zone {"id": "1", "name": "ref-trl-10589-k-Mu22-daman-arora", "uuid": "9145ae72-431c-4e06-a4c5-ab259decf071"}, wait until it is ready to launch console proxy vm
2026-01-13 17:48:23,999 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-8e791e0c]) (logid:78a53f93) System VM Template is not ready at zone [1], wait until it is ready to launch secondary storage VM.
2026-01-13 17:48:53,819 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:[ctx-c576cd8a]) (logid:8eb56943) System vm template is not ready at data center Zone {"id": "1", "name": "ref-trl-10589-k-Mu22-daman-arora", "uuid": "9145ae72-431c-4e06-a4c5-ab259decf071"}, wait until it is ready to launch console proxy vm
2026-01-13 17:48:54,000 DEBUG [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:[ctx-84929584]) (logid:5dcc726d) System VM Template is not ready at zone [1], wait until it is ready to launch secondary storage VM.
darora@dell16plus:~$

  20. Damans227 commented on Jan 13, 2026

    @Damans227
    Damans227
    on Jan 13, 2026
    Collaborator

    Interestingly, it looks like S3 can be used as secondary storage only when it’s region-scoped, not zone-scoped. Even though I tried to create a zone-scoped S3 (at least I thought I was creating one), it actually ended up being created as region-scoped.

    Image

    DefaultEndPointSelector.java line 409 — the bootstrap condition checks for ScopeType.ZONE but S3 is hardcoded to only support ScopeType.REGION in S3ImageStoreProviderImpl.isScopeSupported(). The comment even says "for bootstrap system vm template downloading to region image store" but the code checks for ZONE - so the condition never matches for S3... which may be is the reason that the download of system VM template is never triggered?

  21. nvazquez
    moved this from Discuss to Dev In Progress in Apache CloudStack BugFest - Issueson Jan 14, 2026
  22. Pearl1594
    moved this from Dev In Progress to ready for Review in Apache CloudStack BugFest - Issueson Jan 15, 2026
  23. abh1sar
    modified the milestones: 4.20.3, 4.20.4 on Feb 16, 2026
  24. vishesh92
    linked a pull request that will close this issueFix automatic SystemVM template download to S3 secondary storage #12426on May 20, 2026

  25. github-actions commented on Sep 8, 2026

    @github-actions
    github-actionsBot
    on Sep 8, 2026 – with GitHub Actions

    This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.

  26. github-actions
    added
    no-issue-activity
    on Sep 8, 2026
  27. Damans227
    self-assigned this
    on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:secondary-storagecomponent:systemvm-templateno-issue-activity

Type

Bug

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions