While working on my previous note on how to deploy from Azure DevOps to an AWS EC2 instance, I came across another use case –how to deploy from Azure DevOps to an AWS S3 bucket. This note is about the steps I followed to do so.
Here is a recap so that we’re all on the same page.
Azure DevOps is Microsoft’s solution to a software development process that aids collaboration, traceability, and visibility using components like Azure Boards (work items), Azure Repos (code repository), Azure Pipelines (build and deploy), Azure Artifacts (package management), Azure Test Plans along with a plug and play module to integrate an awesome number of third party tools like Docker, Terraform, etc
AWS S3 stands for simple storage service and as the name suggests it is used to store and retrieve artifacts -files and folder. Here is a link to the official documentation.
Coming back to the use case at hand, I wanted to create a release definition (pipeline) to store artifacts from Azure DevOps into an AWS S3 bucket. This can be broken down into three steps:
Step 1: Create an AWS IAM user with appropriate permissions
Step 2: Create a service connection in Azure DevOps
Step 3: Create a release definition
Please note that there are a couple of pre-requisites to this use case.
-knowledge about AWS IAM -how to create/manage a user
-knowledge about AWS S3 bucket -how to create/manage a bucket
-knowledge about Azure DevOps pipelines -how to create a build and release definition
Step 1: Create an AWS IAM user with appropriate permissions
As we know, to work with resources in AWS, we need appropriate access -read/modify. In this case, we need an IAM user with programmatic access permission (full access) to S3. Please attach appropriate policy (AmazonS3FullAccess) and store the Access key ID and Secret Access key securely. We need those in the next step.
Note: I am purposefully staying out of going into details about IAM -user, group, policy, and role in this post.
Step 2: Create a service connection in Azure DevOps
Now let’s go back to our Azure DevOps portal and launch the project where we want to set up the connection. Click on the gear icon at the bottom left corner. This will bring up the project settings and under Pipelines, we find the Service connections options. If you do not have any existing connections, you’ll get a welcome message.
Click on Create service connection.
This will launch a new side panel with all the new connections available. Chances are that the AWS connection is not available, as I have in the below image.
This means that we’ll have to install the AWS Connection from the marketplace.
Let us close the new service connection panel and return to our project portal. On the top right corner you’ll see a shopping bag icon -Marketplace -> Browse marketplace.
This launches a new page where we may add extensions for our Azure DevOps project. Search for AWS here.
Click on the icon to install the extension to get it for free. Make sure you are logged into the right organization when you are installing this service connection (ensure your email ID displayed at the top is the same email ID tied to your Azure DevOps project).
As instructed on the next page, select the Azure DevOps organization for which you want to install the extension. and click on Install. After installation, you’ll see a message to “Proceed to organization”. Let’s click on that.
Now, lets again navigate to the project and click on the gear icon at the bottom left followed by Service connections and then “Create service connection”. This time we see AWS as the first option to connect to. Yay!
Select AWS and click on Next.
On this page, we are requested to provided mandatory authentication details along with a few optional details. For now, we’ll proceed with Access Key ID and Secret Access Key that we saved from the previous step when we created our IAM user along with the Service Connection name and description.
Once you have the connection created, you’ll see that under Service connections.
That brings us to the end of step 2 and now we proceed with creating a release definition to copy artifacts (files and folders) from Azure DevOps to an S3 bucket in AWS. How exciting!
Step 3: Create a release definition
Let’s navigate back to our project portal by clicking on Overview (vertical right panel) and then click on Pipelines -> Releases.
It’d be good if we already have a Pipeline/Build definition created or some code in the repository that we can upload to S3.
Assuming you have, we proceed to the step when we add a new stage to a release definition.
Click on the + to add a task to agent job and search for S3.
This will list two tasks. We select “Amazon S3 Upload” and click on Add.
Under this task, we will be required to fill details like the AWS Credential, Region, BucketName, and Source Folder. Once that is done, save the release definition and trigger a run.
Note: Here is an image of the task group from a separate project with all values.
After a successful run of the release definition, I was able to view the artifacts in the S3 bucket.
Conclusion: There are multiple options available when it comes to populating AWS S3 bucket from Azure DevOps depending on what the specific use case is. Similar to uploading to S3, we can also download contents from an S3 bucket by using the “Amazon S3 download” task.
As I like to state at the end of my notes, I hope you enjoyed reading the article as much as enjoyed writing it. And if you have any questions or clarifications to seek, please do not hesitate. I will be glad to explore those with you.
Dev Cache 