问题描述
当执行流水线时,执行状态显示失败。之后,在查看此次运行记录的详情时,发现提示 java.lang.OutOfMemoryError: unable to create new native thread 报错。
15:10:47 java.lang.OutOfMemoryError: unable to create new native thread
15:10:47 at java.lang.Thread.start0(Native Method)
15:10:47 at java.lang.Thread.start(Thread.java:717)
15:10:47 at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:95)
15:10:47 at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
15:10:47 at org.jenkinsci.remoting.util.ExecutorServiceUtils.submitAsync(ExecutorServiceUtils.java:59)
15:10:47 at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:63)
15:10:47 at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:93)
15:10:47 at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:45)
15:10:47 at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:306)
15:10:47 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
15:10:47 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
15:10:47 at java.lang.ClassLoader.defineClass1(Native Method)
15:10:47 at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
15:10:47 at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
15:10:47 at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:385)
15:10:47 at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:309)
15:10:47 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
15:10:47 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
15:10:47 at java.lang.Class.forName0(Native Method)
15:10:47 at java.lang.Class.forName(Class.java:264)
15:10:47 at com.sun.proxy.$Proxy9.<clinit>(Unknown Source)
15:10:47 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
15:10:47 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
15:10:47 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
15:10:47 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
15:10:47 at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
15:10:47 at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:169)
15:10:47 at hudson.remoting.Channel.export(Channel.java:811)
15:10:47 at hudson.remoting.Channel.export(Channel.java:774)
15:10:47 at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204)
15:10:47 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15:10:47 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
15:10:47 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15:10:47 at java.lang.reflect.Method.invoke(Method.java:498)
15:10:47 at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230)
15:10:47 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
15:10:47 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
15:10:47 at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
15:10:47 at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
15:10:47 at hudson.remoting.UserRequest.perform(UserRequest.java:222)
15:10:47 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
15:10:47 at hudson.remoting.Request$2.run(Request.java:369)
15:10:47 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
15:10:47 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
15:10:47 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
15:10:47 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
15:10:47 at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
15:10:47 Caused: java.io.IOException: Remote call on a37cc38f-9376-42cb-a959-8813ab1b64d3-6-9hbvx-9tdk8-brg6j failed
15:10:47 at hudson.remoting.Channel.call(Channel.java:1004)
15:10:47 at hudson.FilePath.act(FilePath.java:1069)
15:10:47 at hudson.FilePath.act(FilePath.java:1058)
15:10:47 at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
15:10:47 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:861)
15:10:47 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:833)
15:10:47 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1240)
15:10:47 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
15:10:47 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
15:10:47 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
15:10:47 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
15:10:47 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
15:10:47 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
15:10:47 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
15:10:47 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
15:10:47 at java.lang.Thread.run(Thread.java:748)
15:10:47 Finished: FAILURE
问题原因
在当前承载流水线运行的Kubernetes集群内,节点正在运行的线程数量已达到最大线程数上限,因此,无法创建新的线程,导致流水线执行失败。
解决方案
由于节点支持的最大线程数主要由物理内存决定,所以,可通过使用更高节点配置的Kubernetes集群,达到增加物理内存的目的,从而提升最大线程数上限;或通过扩容当前Kubernetes集群,增加工作节点,分担流水线执行任务。
其中,扩容Kubernetes集群的具体操作步骤,请参考“Kubernetes容器服务”帮助中“扩容集群”的相关内容。