Issue when creating segments / networks on NSX

Hello everyone,

I have a lab with a vCenter / NSX and i am trying to add Cloudstack ontop. I have followed the docs to configure vCenter with NSX and the vCenter cluster has been added to Cloudstack successfully.

Cloudstack version: 4.21.0.0
NSX Version: 4.2.0.1.0.24210154
vCenter Version: 8.0.3

But when i try to create an isolated network the call fails with error:

unable to create NSX network ad00c442-6a64-4772-ab94-43141cc98337 due to: Failed API call to NSX controller

The strange thing is that in NSX the T1 Gateway is being created

Here are the logs from Cloudstack:

2025-10-27 07:10:55,010 DEBUG [o.a.c.s.NsxApiClient] (DirectAgent-22:[ctx-c6ddeea7]) (logid:ea3ad574) NSX Tier-1 gateway with name: D1-A2-Z1-N224 not found
2025-10-27 07:10:55,195 ERROR [o.a.c.s.NsxApiClient] (DirectAgent-161:[ctx-385a40c3]) (logid:ea3ad574) Error creating segment D1-A2-Z1-S224: General error has occurred.
2025-10-27 07:10:55,195 ERROR [o.a.c.r.NsxResource] (DirectAgent-161:[ctx-385a40c3]) (logid:ea3ad574) Failed to create network: adf
2025-10-27 07:10:55,195 ERROR [o.a.c.u.NsxControllerUtils] (qtp1245065720-24:[ctx-d382e352, ctx-25d4fd50]) (logid:ea3ad574) NSX API Command failed
2025-10-27 07:10:55,219 ERROR [c.c.a.ApiServer] (qtp1245065720-24:[ctx-d382e352, ctx-25d4fd50]) (logid:ea3ad574) unhandled exception executing api command: [Ljava.lang.String;@2edc4927 com.cloud.utils.exception.CloudRuntimeException: unable to create NSX network 2b833a44-68a5-4d63-b449-8129c4074289due to: Failed API call to NSX controller
at org.apache.cloudstack.service.NsxGuestNetworkGuru.setup(NsxGuestNetworkGuru.java:153)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.setupNetwork(NetworkOrchestrator.java:838)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$11.doInTransaction(NetworkOrchestrator.java:3098)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$11.doInTransaction(NetworkOrchestrator.java:2995)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2995)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createGuestNetwork(NetworkOrchestrator.java:2734)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:2332)
at com.cloud.network.NetworkServiceImpl$4.doInTransaction(NetworkServiceImpl.java:2270)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.network.NetworkServiceImpl.commitNetwork(NetworkServiceImpl.java:2270)
at com.cloud.network.NetworkServiceImpl.createGuestNetwork(NetworkServiceImpl.java:1830)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy200.createGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.CreateNetworkCmd.execute(CreateNetworkCmd.java:431)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:854)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:668)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:416)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:193)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:190)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:149)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:840)

I have also tried to create a VPC which works ok and i can ping the public IP address it allocates but when i try to create a network inside the VPC it fails again with the same error.

So my guess is that there is something going on when Cloudstack tries to create a segment on NSX.

Is this a bug or have i forgotten to configure something?

Tái hiện việc tạo mạng cô lập trên các phiên bản CloudStack, NSX và vCenter được nêu, sau đó lần theo lỗi từ NsxGuestNetworkGuru.java:153 qua NsxResource và NsxApiClient, sử dụng các mục nhật ký tạo segment làm điểm bắt đầu. So sánh phản hồi của NSX API với yêu cầu được gửi cho segment; công việc được xem là hoàn tất khi xác định được nguyên nhân và việc tạo mạng либо thành công hoặc báo cáo một lỗi cụ thể có thể xử lý.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.