Inaccessible Jenkins Server

[gruntwork-support]

This message was extracted from a discussion that originally took place in Gruntwork Community Slack. Names and URLs have been removed where appropriate

From a customer

We've built Jenkins AMI using terragrunt module with ref v0.44.7 using packer.
When we try to create Jenkins server using terragrunt module, after creating LB and EC2 instance, health check was keep failing. Hence EC2 keep getting restarted. If we manually start the Jenkins server from the EC2 instance then it is working. But if reboot the instance then getting same problem i.e jenkins server not automatically starting up within the EC2 instance. The same behavior occurred both versions terraform-aws-service-catalog.git//modules/mgmt/jenkins?ref=v0.34.1 and terraform-aws-service-catalog.git//modules/mgmt/jenkins?ref=v0.44.7.
Any idea why Jenkins server not starting up automatically once EC2 instance is successfully launched?

[gruntwork-support]

From a grunt

Hi person, I will be investigating this today. I am not sure what is causing this behavior off the top of my head, and will require some digging and experimentation to identify the problem. I will provide an update on the investigation by this afternoon.

[gruntwork-support]

From a customer

Thanks person.

FYI, I ssh to the instance and when I checked status getting this.

Attached File

[gruntwork-support]

From a grunt

Something to help: Could this be due to a missing IAM Group?

user-data: Starting ssh-grunt
start-amazon-cloudwatch-agent[1151]: Valid Json input schema.
start-amazon-cloudwatch-agent[1151]: I! Detecting run_as_user...
user-data: ssh-grunt 2021/07/08 11:47:39 NoSuchEntity: The group with name ssh-grunt-sudo-users cannot be found.

After that error, I think it fails to finish the Execute cloud user/final scripts:

cloud-init[923]: Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~20.04.1 running 'modules:final' at Thu, 08 Jul 2021 11:47:23 +0000. Up 62.89 seconds.
cloud-init[923]: 2021-07-08 11:47:39,380 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
cloud-init[923]: 2021-07-08 11:47:39,383 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
...
systemd[1]: cloud-final.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: cloud-final.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Execute cloud user/final scripts.

(based on attached logs)

[gruntwork-support]

From a customer

I don't think so, Actually once I remove from ASG, then EC2 instance running properly. The main issue here is jenkins is not starting up. When I started using this command sudo service jenkins status then it starts able to attach it to ALB. But looking at the script from terragrunt module , It is using following command.

run-jenkins --memory 1024m --jenkins-home /jenkins

From TF, we are passing memory value as 1024m and mount path is default set to /jenkins

If I try the same way from Jenkins EC2 instance , I'm getting this error.

run-jenkins --memory 1024m --jenkins-home /jenkins
Setting JVM settings in /etc/default/jenkins to: JAVA_ARGS="-server -Xms1024m -Xmx1024m -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -Djava.awt.headless=true "
Configuring Jenkins home dir as /jenkins
Starting Jenkins
Job for jenkins.service failed because the control process exited with error code

systemctl status jenkins.service
● jenkins.service - LSB: Start Jenkins at boot time
     Loaded: loaded (/etc/init.d/jenkins; generated)
     Active: failed (Result: exit-code) since Thu 2021-07-08 14:48:44 UTC; 7min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 1767 ExecStart=/etc/init.d/jenkins start (code=exited, status=7)

Jul 08 14:48:42 ip-172-31-33-1 systemd[1]: Starting LSB: Start Jenkins at boot time...
Jul 08 14:48:43 ip-172-31-33-1 jenkins[1767]: Correct java version found
Jul 08 14:48:43 ip-172-31-33-1 jenkins[1767]:  * Starting Jenkins Automation Server jenkins
Jul 08 14:48:43 ip-172-31-33-1 su[1800]: (to jenkins) root on none
Jul 08 14:48:43 ip-172-31-33-1 su[1800]: pam_unix(su-l:session): session opened for user jenkins by (uid=0)
Jul 08 14:48:43 ip-172-31-33-1 su[1800]: pam_unix(su-l:session): session closed for user jenkins
Jul 08 14:48:44 ip-172-31-33-1 jenkins[1767]:    ...fail!

[gruntwork-support]

From a grunt

Ok after investigation, what person said is correct. The main issue here is that you have ssh-grunt configured for the server, but it looks like your environment is not setup to support it.

Because the boot script fails to start up ssh-grunt, everything after that step fails, including mounting the data volume. The reason directly starting the jenkins service works is because the default location for the jenkins volume is /var/lib, which exists in the file system. When you call run-jenkins, you are configuring the jenkins data volume to be /jenkins, which does not exist by default - this materializes when the boot script runs attach_volume, which isn’t called if it halts when it tries to start ssh-grunt.

You can workaround this by disabling ssh-grunt. This can be done by setting the following terraform input vars:

enable_ssh_grunt = false
ssh_grunt_iam_group = ""
ssh_grunt_iam_group_sudo = ""

[gruntwork-support]

From a customer

Yes, I ran TF apply but didn't manually rotate the instance.

... sharing logs ...

This the logs after rotating the instance.

Let me destroy and re-run it.

[gruntwork-support]

From a grunt

Thanks for sharing the logs. It looks like it got all the way to run-jenkins, but I suspect it is running into the OOM issue. As a temporary workaround, does it work if you bump the instance type to t3.large?
Separately, I am investigating why it is getting OOM as it should have enough memory on the t3.medium. However, it's taking a bit for me to create a repro instance.

Ok after some investigation, I am unable to reproduce this issue. That is, I can get jenkins to come up successfully without hitting any memory limits when trying with t3.medium and 1024m of memory.

Did you make any modifications to the packer template?

[gruntwork-support]

From a customer

Nope. Actually I'm able to get jenkins server deployed in a separate VPC with IGW attached. However getting same issue with VPC with no IGW or NAT-gatway.

[gruntwork-support]

From a grunt

Ah I think you will need to configure some kind of NAT for jenkins, as many routines are going to need outbound access. At a minimum, the jenkins server needs to be able to communicate with the AWS API so that it can mount the EBS volume on every reboot.

[gruntwork-support]

From a customer

Exactly. Thanks person , I think at least I'm clear now it is nothing to do AMI image .