At Logic Forte, our CI/CD pipelines typically use AWS CodeBuild to pull Git repositories and build/test/deploy Docker images. Our typical build will pull a public image from Docker Hub, build a custom image, and then save our custom image to a private repo on ECR for testing/deployment.
Docker has been notifying users that they would begin rate limiting public image requests in November 2020. The new limits of 100 pulls per hour may seem like it would not affect your occasional builds, especially if you deploy from a private repo like ECR.
Nearly all of our image pulls use our private repo when we deploy and when we scale up/down. We might build 5-10 times per day, which is far below the new 100 pulls per hour limit. However, if you use a cloud service (such as CodeBuild) to build your Docker images, this new rate limiting will likely affect you since the cloud build service will likely exceed Docker’s new rate limit thresholds. The service is building images for thousands of other customers, so Docker seems thousands of pull requests coming from the build service and they no idea who the requests belong to.
Our CodePipeline failed immediately after Docker began enforcing the new rate limits. Specifically, our CodeBuild job failed with the following errors:
Building the Docker image... [Container] 2020/11/04 18:19:25 Running command docker build -t $REPOSITORY_URI:$IMAGE_TAG . Sending build context to Docker daemon 6.797MB Step 1/17 : FROM php:7.3-apache toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit [Container] 2020/11/04 18:19:26 Command did not exit successfully docker build -t $REPOSITORY_URI:$IMAGE_TAG . exit status 1 [Container] 2020/11/04 18:19:26 Phase complete: BUILD State: FAILED
Fortunately, the solution to the “toomanyrequests: You have reached your pull rate limit” error was simple and straightforward.
- Create account at Docker.com. We created a new FREE individual account at Docker.com just for authenticating pull requests.
- Create a new Access Token in your Docker account under Account Settings then Security. We named our access token “AWS CodePipeline”.
- Update your project
buildspec.ymlfile used by CodeBuild so that it authenticates to Docker prior to building images. We added the following 3 lines to our build spec, immediately before our “docker build” command. Substitute “mybucket”, “mypath”, and “myusername” with valid values for your environment.
aws s3 cp s3://mybucket/mypath/.docker.key .docker.key docker login -u myusername --password-stdin < .docker.key rm .docker.key
To be clear, we placed our Docker Access Token in a file named “.docker.key” and uploaded the “.docker.key” file to a restricted S3 bucket. Our build process is already reading other files from the S3 bucket, so adding this file to the mix was easy. If your CodeBuild job is not already interacting with S3, you will need to setup IAM permissions to allow CodeBuild to read from a bucket. An alternate solution would be to store your Access Token in AWS Secrets Manager and retrieve the secret token during the build process.
Need a quick fix? You could temporarily use the following single line solution in your Build Spec file (instead of the 3 lines we are using above) to test and ensure that authenticating to Docker resolves your build errors.
docker login -u myusername -p myaccesstoken
Be aware that storing your Access Token in your Build Spec file is not considered best practice, since your access token would end up being committed to a shared code repository. If you test with this one-liner, be sure to remove the access token from your Docker account after you are done testing!
Please reply below if you found this helpful. It is always nice to hear that these posts are helping others. 🙂