AWS CodeBuild Failed due to Docker pull rate limit. Solution: Update buildspec.yml file.

At Logic Forte, our CI/CD pipelines typically use AWS CodeBuild to pull Git repositories and build/test/deploy Docker images. Our typical build will pull a public image from Docker Hub, build a custom image, and then save our custom image to a private repo on ECR for testing/deployment.

Docker has been notifying users that they would begin rate limiting public image requests in November 2020. The new limits of 100 pulls per hour may seem like it would not affect your occasional builds, especially if you deploy from a private repo like ECR.

Nearly all of our image pulls use our private repo when we deploy and when we scale up/down. We might build 5-10 times per day, which is far below the new 100 pulls per hour limit. However, if you use a cloud service (such as CodeBuild) to build your Docker images, this new rate limiting will likely affect you since the cloud build service will likely exceed Docker’s new rate limit thresholds. The service is building images for thousands of other customers, so Docker seems thousands of pull requests coming from the build service and they no idea who the requests belong to.

Problem

Our CodePipeline failed immediately after Docker began enforcing the new rate limits. Specifically, our CodeBuild job failed with the following errors:

Building the Docker image...

[Container] 2020/11/04 18:19:25 Running command docker build -t $REPOSITORY_URI:$IMAGE_TAG .
Sending build context to Docker daemon  6.797MB
Step 1/17 : FROM php:7.3-apache
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[Container] 2020/11/04 18:19:26 Command did not exit successfully docker build -t $REPOSITORY_URI:$IMAGE_TAG . exit status 1
[Container] 2020/11/04 18:19:26 Phase complete: BUILD State: FAILED

Solution

Fortunately, the solution to the “toomanyrequests: You have reached your pull rate limit” error was simple and straightforward.

  1. Create account at Docker.com. We created a new FREE individual account at Docker.com just for authenticating pull requests.
  2. Create a new Access Token in your Docker account under Account Settings then Security. We named our access token “AWS CodePipeline”.
  3. Update your project buildspec.yml file used by CodeBuild so that it authenticates to Docker prior to building images. We added the following 3 lines to our build spec, immediately before our “docker build” command. Substitute “mybucket”, “mypath”, and “myusername” with valid values for your environment.
aws s3 cp s3://mybucket/mypath/.docker.key .docker.key
docker login -u myusername --password-stdin < .docker.key
rm .docker.key

To be clear, we placed our Docker Access Token in a file named “.docker.key” and uploaded the “.docker.key” file to a restricted S3 bucket. Our build process is already reading other files from the S3 bucket, so adding this file to the mix was easy. If your CodeBuild job is not already interacting with S3, you will need to setup IAM permissions to allow CodeBuild to read from a bucket. An alternate solution would be to store your Access Token in AWS Secrets Manager and retrieve the secret token during the build process.

Need a quick fix? You could temporarily use the following single line solution in your Build Spec file (instead of the 3 lines we are using above) to test and ensure that authenticating to Docker resolves your build errors.

docker login -u myusername -p myaccesstoken

Be aware that storing your Access Token in your Build Spec file is not considered best practice, since your access token would end up being committed to a shared code repository. If you test with this one-liner, be sure to remove the access token from your Docker account after you are done testing!

Please reply below if you found this helpful. It is always nice to hear that these posts are helping others. 🙂

4 thoughts on “AWS CodeBuild Failed due to Docker pull rate limit. Solution: Update buildspec.yml file.

  1. Thanks, very useful. I much prefer using the parameter store (for which I already had permissions set up) though. A small update to the `buildspec`:

    “`
    env:
    parameter-store:
    DOCKERHUB_USER: /CodeBuild/DOCKERHUB_USER
    DOCKERHUB_PASSWORD: /CodeBuild/DOCKERHUB_PASSWORD

    phases:
    pre_build:
    commands:
    – echo Logging in to docker hub…
    – docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASSWORD

    “`

  2. This was really a savior post which helped me implement a strategy to deal with docker rate limit. I went ahead with storing docker username and password in AWS secret manager and then retrieved in buildspec.yml

Leave a Reply

Your email address will not be published. Required fields are marked *