Learn how to – Build AWS EC2 Machine Images (AMI) With Packer and Ansible.
If you’re running your Production Virtual machines in AWS, chances are you’ve encountered scenarios where you need to build your custom Amazon Machine Images (AMI). An AMI provides the information required to launch an instance, which may include Base Operating system, application dependencies, and other runtime libraries required. Multiple instances can be started from a single AMI with the same configuration.
What is Packer?
Packer is an open source tool used to create machine images for multiple platforms from a single source configuration. It gives you the flexibility of building your custom AMI for use in AWS EC2 platform.
Ansible in the mix?
Ansible configuration management tool will be used during the build process to setup application environment, dependencies and even deploy an Application into the AMI. This process can be fully automated for integrating into CI/CD pipeline.
In this tutorial, we will consider an example, which builds an AMI using Packer and Ansible. We will use a simple Java Web Application (WAR) for demonstration. Ansible will be used to install Java Web Application dependencies and to deploy the WAR file.
Step 1: Setup Dependencies
- A Linux/macOS system to work on
- Install Ansible
To install Ansible, use the following commands.
###### CentOS / Fedora / RHEL ######
sudo yum -y install epel-release
wget https://bootstrap.pypa.io/pip/3.6/get-pip.py
sudo python3 get-pip.py
sudo pip3 install ansible
###### Ubuntu/Debian/Linux Mint ######
sudo apt update
sudo apt install -y software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt update
sudo apt install -y ansible
###### Arch/Manjaro ######
sudo pacman -S ansible
###### macOS ######
sudo easy_install pip
sudo pip install ansible
- Install Packer – Install Latest Packer on Linux / FreeBSD / macOS / Windows
- Install and configure AWS CLI Tool: How to Install and Use AWS CLI on Linux
Don’t forget to configure AWS Access Key ID and Secret Access Key as shown on the installation guide.
Once all the pre-requisite software are installed, proceed to the next sections.
Step 2: Create a Project Skeleton
Let’s create a directory for our project.
mkdir -p ~/projects/packer-ansible-aws
cd ~/projects/packer-ansible-awsUnder created directory, create folders for Packer, Ansible provisioners and where the Application source code/build packages are placed.
mkdir -p ~/projects/packer-ansible-aws/packer/provisioners/{ansible,scripts}
mkdir -p ~/projects/packer-ansible-aws/packer/provisioners/ansible/{templates,files}
mkdir -p ~/projects/packer-ansible-aws/src/application
This is my initial project tree.
$ cd ~/projects/packer-ansible-aws
$ tree
.
├── packer
│ └── provisioners
│ ├── ansible
│ └── scripts
└── src
└── application
6 directories, 0 filesStep 3: Create Packer Templates
We can now create a packer json file that will be used to build an AMI image. Inside Packer file are keys defined. We’ll use variables, builders, and provisioners.
Create a Packer template to be used.
cd packer
vim packer-build.jsonMy contents looks like below.
{
"variables": {
"aws_access_key": "",
"aws_secret_key": "",
"ami_name": "tomcat-ami",
"aws_region": "us-east-1",
"ssh_username": "centos",
"vpc_id": "",
"subnet_id": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"instance_type": "t2.micro",
"force_deregister": "true",
"ssh_username": "{{user `ssh_username`}}",
"communicator": "ssh",
"associate_public_ip_address": true,
"subnet_id": "{{user `subnet_id`}}",
"ami_name": "{{user `ami_name`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "CentOS Linux 7 x86_64 HVM EBS *",
"root-device-type": "ebs"
},
"owners": ["679593333241"],
"most_recent": true
},
"run_tags": {
"Name": "packer-build-image"
}
}],
"provisioners": [{
"type": "shell",
"inline": "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
},
{
"type": "shell",
"script": "./provisioners/scripts/bootstrap.sh"
},
{
"type": "ansible",
"playbook_file": "./provisioners/ansible/setup-server.yml"
},
{
"type": "ansible",
"playbook_file": "./provisioners/ansible/deploy_app.yml"
}]
}
Under variables key section, set required variables:
- aws_access_key & aws_secret_key – Ignore this if configured in the AWS CLI tool installation section. Often set on ~/.aws/credentials
- ami_name – Name to be given to AMI generated by Packer
- aws_region – Region where Temporary instance will be created and created AMI stored.
- ssh_username – AMI SSH user. Since I’m using CentOS 7 image available on AWS as a base image, the default ssh user is centos, for Ubuntu, use ubuntu.
- vpc_id & subnet_id – The VPC ID and the Subnet ID to be used by a temporary instance created by the packer. It needs to be accessible from the workstation machine. I recommend you use public subnet.
Under builders key section, set:
- instance_type – The EC2 instance type to use while building the AMI.
- source_ami_filter: The initial AMI used as a base for the newly created machine image. Its value can be AMI ID or a filter to get ID.
Consult the AMI Builder documentation for more details.
On provisioners section, provide the paths to your Bash scripts and Ansible roles to be executed during build.
Step 4: Create Scripts & Ansible Playbooks
Let’s start with a playbook that will prepare a CentOS server for hosting a Java Web Application.
We’ll install Ansible on the VM using bash script which runs before Ansible playbooks are executed.
$ vim ./provisioners/scripts/bootstrap.sh
#!/bin/bash
set -ex
# Add EPEL repository
sudo yum install -y epel-release
sudo yum install -y ansibleMake the script executable:
chmod +x ./provisioners/scripts/bootstrap.shCreate a file named setup-server.yml inside provisioners/ansible directory.
vim ~/projects/packer-ansible-aws/packer/provisioners/ansible/setup-server.ymlThe contents for the file are:
---
- name: Tomcat deployment playbook
hosts: 'all'
become: yes
become_method: sudo
tasks:
- name: Add EPEL repository
yum:
name: epel-release
state: present
- name: Update all packages
yum:
name: "*"
state: latest
- name: Install basic packages
yum:
name: ['epel-release', 'firewalld', 'vim', 'bash-completion', 'htop', 'tmux', 'screen', 'telnet', 'tree', 'wget', 'curl', 'git', 'python-firewall']
state: present
- name: Install Java
yum:
name: java-1.8.0-openjdk
state: present
- name: Add tomcat group
group:
name: tomcat
- name: Add "tomcat" user
user:
name: tomcat
group: tomcat
home: /usr/share/tomcat
createhome: no
system: yes
- name: Download Tomcat
get_url:
url: https://archive.apache.org/dist/tomcat/tomcat-9/v9.0.75/bin/apache-tomcat-9.0.75.tar.gz
dest: /tmp/apache-tomcat-9.0.75.tar.gz
- name: Create a tomcat directory
file:
path: /usr/share/tomcat
state: directory
owner: tomcat
group: tomcat
- name: Extract tomcat archive
unarchive:
src: /tmp/apache-tomcat-9.0.75.tar.gz
dest: /usr/share/tomcat
owner: tomcat
group: tomcat
remote_src: yes
extra_opts: "--strip-components=1"
creates: /usr/share/tomcat/bin
- name: Copy tomcat service file
template:
src: templates/tomcat.service.j2
dest: /etc/systemd/system/tomcat.service
- name: Start and enable tomcat
service:
daemon_reload: yes
name: tomcat
state: started
enabled: yes
- name: Start and enable firewalld
service:
name: firewalld
state: started
enabled: yes
- name: Open tomcat port on the firewall
firewalld:
port: 8080/tcp
permanent: true
state: enabled
immediate: yes
when: "ansible_os_family == 'RedHat' and ansible_distribution_major_version == '7'"
handlers:
- name: restart tomcat
service:
name: tomcat
state: restartedWe also need to add Tomcat systemd service as template.
cd ~/projects/packer-ansible-aws/packer/provisioners/ansible/templates/
vim tomcat.service.j2Paste below contents into the file.
[Unit]
Description=Tomcat
After=syslog.target network.target
[Service]
Type=forking
User=tomcat
Group=tomcat
Environment=JAVA_HOME=/usr/lib/jvm/jre
Environment='JAVA_OPTS=-Djava.awt.headless=true'
Environment=CATALINA_HOME=/usr/share/tomcat
Environment=CATALINA_BASE=/usr/share/tomcat
Environment=CATALINA_PID=/usr/share/tomcat/temp/tomcat.pid
Environment='CATALINA_OPTS=-Xms256M -Xmx512M'
ExecStart=/usr/share/tomcat/bin/catalina.sh start
ExecStop=/usr/share/tomcat/bin/catalina.sh stop
[Install]
WantedBy=multi-user.targetThese ansible playbook will:
- Install EPEL repository on CentOS 7 VM created by Packer
- Update all system packages to latest releases
- Install OS basic packages – vim, firewalld, wget e.t.c.
- Install Java 8
- Download, install and configure Tomcat 8.5.x
- Start tomcat service and configure firewalld
The next playbook will deploy Sample Java Application packaged as war file and can be downloaded here.
So let’s start by downloading the sample war Application from Tomcat website.
cd ~/projects/packer-ansible-aws/packer/provisioners/ansible/files
wget https://tomcat.apache.org/tomcat-9.0-doc/appdev/sample/sample.warThen create a playbook to deploy War Application to AMI to be created.
vim ~/projects/packer-ansible-aws/packer/provisioners/ansible/deploy_app.ymlThe data to be populated is:
---
- name: Deploy tomcat war application
hosts: 'all'
become: yes
become_method: sudo
tasks:
- name: Deploy war file to tomcat
copy:
src: files/sample.war
dest: /usr/share/tomcat/webapps/sample.war
owner: tomcat
mode: 0644
notify: restart tomcat
handlers:
- name: restart tomcat
service:
name: tomcat
state: restartedStep 5: Run Packer build
Let’s now build the AMI and save results to build_artifact.txt file
cd ~/projects/packer-ansible-aws/packer
packer build -machine-readable packer-build.json | tee build_artifact.txtSample output:
...................................................
1559285322,,ui,say,==> amazon-ebs: Force Deregister flag found%!(PACKER_COMMA) skipping prevalidating AMI Name
1559285327,,ui,message, amazon-ebs: Found Image ID: ami-02eac2c0129f6376b
1559285330,,ui,say,==> amazon-ebs: Creating temporary keypair: packer_5cf0ce47-7f3e-ded7-9053-0732a3789020
1559285332,,ui,say,==> amazon-ebs: Creating temporary security group for this instance: packer_5cf0ce54-366a-3221-906e-cd3f1af3c1ee
1559285335,,ui,say,==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
1559285337,,ui,say,==> amazon-ebs: Launching a source AWS instance...
1559285337,,ui,say,==> amazon-ebs: Adding tags to source instance
1559285337,,ui,message, amazon-ebs: Adding tag: "Name": "packer-build-image"
1559285339,,ui,message, amazon-ebs: Instance ID: i-0ffaa1eef9ea7966b
1559285339,,ui,say,==> amazon-ebs: Waiting for instance (i-0ffaa1eef9ea7966b) to become ready...
1559285374,,ui,say,==> amazon-ebs: Using ssh communicator to connect: 35.172.100.134
1559285374,,ui,say,==> amazon-ebs: Waiting for SSH to become available...
1559285449,,ui,say,==> amazon-ebs: Connected to SSH!
1559285449,,ui,say,==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell051244737
1559285456,,ui,say,==> amazon-ebs: Provisioning with shell script: ./provisioners/scripts/bootstrap.sh
1559285460,,ui,error,==> amazon-ebs: + sudo yum install -y epel-release
................................................
1559285482,,ui,say,==> amazon-ebs: Provisioning with Ansible...
1559285483,,ui,say,==> amazon-ebs: Executing Ansible: ansible-playbook --extra-vars packer_build_name=amazon-ebs packer_builder_type=amazon-ebs -o IdentitiesOnly=yes -i /tmp/packer-provisioner-ansible406384377 /home/jmutai/projects/packer-ansible-aws/packer/provisioners/ansible/setup-server.yml -e ansible_ssh_private_key_file=/tmp/ansible-key463353218
1559285483,,ui,message, amazon-ebs:
1559285483,,ui,message, amazon-ebs: PLAY [Tomcat deployment playbook] **********************************************
1559285483,,ui,message, amazon-ebs:
1559285483,,ui,message, amazon-ebs: TASK [Gathering Facts] *********************************************************
1559285508,,ui,message, amazon-ebs: ok: [default]
1559285508,,ui,message, amazon-ebs:
1559285508,,ui,message, amazon-ebs: TASK [Add EPEL repository] *****************************************************
1559285553,,ui,message, amazon-ebs: ok: [default]
1559285553,,ui,message, amazon-ebs:
1559285553,,ui,message, amazon-ebs: TASK [Update all packages] *****************************************************
1559285564,,ui,message, amazon-ebs: ok: [default]
1559285564,,ui,message, amazon-ebs:
1559285564,,ui,message, amazon-ebs: TASK [Install basic packages] **************************************************
1559285595,,ui,message, amazon-ebs: changed: [default]
1559285595,,ui,message, amazon-ebs:
1559285595,,ui,message, amazon-ebs: TASK [Install Java] ************************************************************
1559285630,,ui,message, amazon-ebs: changed: [default]
1559285630,,ui,message, amazon-ebs:
1559285630,,ui,message, amazon-ebs: TASK [Add tomcat group] ********************************************************
1559285640,,ui,message, amazon-ebs: changed: [default]
1559285640,,ui,message, amazon-ebs:
1559285640,,ui,message, amazon-ebs: TASK [Add "tomcat" user] *******************************************************
1559285659,,ui,message, amazon-ebs: changed: [default]
1559285659,,ui,message, amazon-ebs:
1559285659,,ui,message, amazon-ebs: TASK [Download Tomcat] *********************************************************
1559285674,,ui,message, amazon-ebs: changed: [default]
1559285674,,ui,message, amazon-ebs:
1559285674,,ui,message, amazon-ebs: TASK [Create tomcat directory] *************************************************
1559285692,,ui,message, amazon-ebs: changed: [default]
1559285692,,ui,message, amazon-ebs:
1559285692,,ui,message, amazon-ebs: TASK [Extract tomcat archive] **************************************************
1559285724,,ui,message, amazon-ebs: changed: [default]
1559285724,,ui,message, amazon-ebs:
1559285724,,ui,message, amazon-ebs: TASK [Copy tomcat service file] ************************************************
1559285752,,ui,message, amazon-ebs: changed: [default]
1559285752,,ui,message, amazon-ebs:
1559285752,,ui,message, amazon-ebs: TASK [Start and enable tomcat] *************************************************
1559285772,,ui,message, amazon-ebs: changed: [default]
1559285772,,ui,message, amazon-ebs:
1559285772,,ui,message, amazon-ebs: TASK [Start and enable firewalld] **********************************************
1559285783,,ui,message, amazon-ebs: changed: [default]
1559285783,,ui,message, amazon-ebs:
1559285783,,ui,message, amazon-ebs: TASK [Open tomcat port on firewall] ********************************************
1559285796,,ui,message, amazon-ebs: changed: [default]
1559285796,,ui,message, amazon-ebs:
1559285796,,ui,message, amazon-ebs: PLAY RECAP *********************************************************************
1559285796,,ui,message, amazon-ebs: default : ok=14 changed=11 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
1559285796,,ui,message, amazon-ebs:
1559285796,,ui,say,==> amazon-ebs: Provisioning with Ansible...
1559285797,,ui,say,==> amazon-ebs: Executing Ansible: ansible-playbook --extra-vars packer_build_name=amazon-ebs packer_builder_type=amazon-ebs -o IdentitiesOnly=yes -i /tmp/packer-provisioner-ansible221665956 /home/jmutai/projects/packer-ansible-aws/packer/provisioners/ansible/deploy_app.yml -e ansible_ssh_private_key_file=/tmp/ansible-key672089113
1559285797,,ui,message, amazon-ebs:
1559285797,,ui,message, amazon-ebs: PLAY [Deploy tomcat war application] *******************************************
1559285797,,ui,message, amazon-ebs:
1559285797,,ui,message, amazon-ebs: TASK [Gathering Facts] *********************************************************
1559285811,,ui,message, amazon-ebs: ok: [default]
1559285811,,ui,message, amazon-ebs:
1559285811,,ui,message, amazon-ebs: TASK [Deploy war file to tomcat] ***********************************************
1559285832,,ui,message, amazon-ebs: changed: [default]
1559285832,,ui,message, amazon-ebs:
1559285832,,ui,message, amazon-ebs: RUNNING HANDLER [restart tomcat] ***********************************************
1559285846,,ui,message, amazon-ebs: changed: [default]
1559285846,,ui,message, amazon-ebs:
1559285846,,ui,message, amazon-ebs: PLAY RECAP *********************************************************************
1559285846,,ui,message, amazon-ebs: default : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0Once done with provisioning, packer will Stop and destroy temporary instance used, then create an AMI.
1559285846,,ui,message, amazon-ebs:
1559285846,,ui,say,==> amazon-ebs: Stopping the source instance...
1559285846,,ui,message, amazon-ebs: Stopping instance
1559285850,,ui,say,==> amazon-ebs: Waiting for the instance to stop...
1559285875,,ui,say,==> amazon-ebs: Creating AMI tomcat-ami from instance i-0ffaa1eef9ea7966b
1559285884,,ui,message, amazon-ebs: AMI: ami-0f6cc044e485adabf
1559285884,,ui,say,==> amazon-ebs: Waiting for AMI to become ready...
1559285967,,ui,say,==> amazon-ebs: Terminating the source AWS instance...
1559285986,,ui,say,==> amazon-ebs: Cleaning up any extra volumes...
1559285987,,ui,say,==> amazon-ebs: Destroying volume (vol-0e34552c157dc217f)...
1559285988,,ui,say,==> amazon-ebs: Deleting temporary security group...
1559285990,,ui,say,==> amazon-ebs: Deleting temporary keypair...
1559285991,,ui,say,Build 'amazon-ebs' finished.
1559285991,,ui,say,n==> Builds finished. The artifacts of successful builds are:
1559285991,amazon-ebs,artifact-count,1
1559285991,amazon-ebs,artifact,0,builder-id,mitchellh.amazonebs
1559285991,amazon-ebs,artifact,0,id,us-east-1:ami-0f6cc044e485adabf
1559285991,amazon-ebs,artifact,0,string,AMIs were created:nus-east-1: ami-0f6cc044e485adabfn
1559285991,amazon-ebs,artifact,0,files-count,0
1559285991,amazon-ebs,artifact,0,end
1559285991,,ui,say,--> amazon-ebs: AMIs were created:nus-east-1: ami-0f6cc044e485adabfnAll provisioning output will be written to the build_artifact.txt file inside packer folder. AMI ID is printed at the end – ami-0f6cc044e485adabf.
Step 6: Testing AMI Created
In this section, I’ll use Terraform to provision a new instance with created AMI. The same can be done from AWS console.
See how to install Terraform on Linux. Check releases page for the latest version.
cd /tmp
export VER="1.4.6"
wget https://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip
unzip terraform_${VER}_linux_amd64.zip
sudo mv terraform /usr/local/bin/
terraform -vCreate terraform projects directory.
mkdir ~/projects/packer-ansible-aws/terraform
cd ~/projects/packer-ansible-aws/terraform
vim main.tfMy main.tf terraform file looks like this.
# Provider
provider "aws" {
region = "us-east-1"
}
# Create EC2 Test instance
resource "aws_instance" "test-instance" {
key_name = ""
subnet_id = ""
ami = ""
instance_type = "t2.micro"
associate_public_ip_address = "true"
disable_api_termination = "false"
monitoring = "false"
vpc_security_group_ids = ["${aws_security_group.test-sg.id}"]
tags {
Name = "test-instance"
}
}
# Create Test SG
resource "aws_security_group" "test-sg" {
vpc_id = ""
name = "test-sg"
description = "Test Security group"
tags {
Name = "test-sg"
}
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 0
to_port = 65535
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = -1
to_port = -1
protocol = "icmp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}Set all values required then initialize a Terraform working directory
$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.1.0...
- Installed hashicorp/aws v5.1.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.Show an execution plan.
$ terraform plan
.......................
Plan: 2 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------Finally, build your infrastructure according to Terraform configuration files in DIR.
$ terraform apply
......................
Plan: 2 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
aws_security_group.test-sg: Creating...
arn: "" => "<computed>"
description: "" => "Test Security group"
egress.#: "" => "1"
....................................................................
aws_security_group.test-sg: Still creating... (10s elapsed)
aws_security_group.test-sg: Creation complete after 11s (ID: sg-062a8615dd461fc1e)
aws_instance.test-instance: Creating...
..............................................
aws_instance.test-instance: Still creating... (10s elapsed)
aws_instance.test-instance: Still creating... (20s elapsed)
aws_instance.test-instance: Still creating... (30s elapsed)
aws_instance.test-instance: Still creating... (40s elapsed)
aws_instance.test-instance: Creation complete after 45s (ID: i-0bf06f3b3cbf99791)
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.The new instance created can be confirmed from AWS EC2 dashboard.
Get instance IP and access service port. Test Java Application should be accessible on:
http://serverip:8080/sample/
Similar to this:
To destroy your test infrastructure, run:
$ terraform destroy
aws_security_group.test-sg: Refreshing state... (ID: sg-062a8615dd461fc1e)
aws_instance.test-instance: Refreshing state... (ID: i-0bf06f3b3cbf99791)
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- aws_instance.test-instance
- aws_security_group.test-sg
Plan: 0 to add, 0 to change, 2 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yesYou should see
aws_instance.test-instance: Destroying... (ID: i-0bf06f3b3cbf99791)
aws_instance.test-instance: Still destroying... (ID: i-0bf06f3b3cbf99791, 10s elapsed)
aws_instance.test-instance: Still destroying... (ID: i-0bf06f3b3cbf99791, 20s elapsed)
aws_instance.test-instance: Destruction complete after 25s
aws_security_group.test-sg: Destroying... (ID: sg-062a8615dd461fc1e)
aws_security_group.test-sg: Destruction complete after 2s
Destroy complete! Resources: 2 destroyed.You have learned how to use Packer and Ansible to create AWS AMI. Stay connected for more cool tutorials.
Similar:
Best Storage Solutions for Kubernetes & Docker Containers
How to reset / change IAM user password on AWS
How to Reset RDS Master User Password on AWS
How to extend EBS boot disk on AWS without an instance reboot
How to Provision VMs on KVM with Terraform
Leave a Reply
You must be logged in to post a comment.