Boto3 — AWS command lines

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#s3

EC2

Create EC2

ssh using your key pair

Go to folder that had key pair:

Connect to EC2:

ssh -i “keypairname” ec2-user@aws ec2 name.compute-1.amazonaws.com

You are now connected to ec2

• ** your ssh command is also present in the EC2 connect button

Install python in EC2 using terminal

should be in ec2 location

[ec2-user@ip-142–84–78–4 ~]$ — something like this

sudo yum install python3 -y

python — — version

Other installs in EC2 —

if you are running a python script that calls libraries then they all need to be installed in EC2

sudo pip3 install tweepy

sudo pip3 install boto3

sudo pip install sklearn

Configure AWS on your local folder

When you set up your amazon account you get this

On your terminal

aws configure

AWS Access Key ID [****************AVGA]: Enter key here

AWS Secret Access Key [****************BYKx]: Enter Secret key

Default region name [us-east-1]: us-east-1

Default output format [json]: jason

Boto3

Boto is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services.

Boto3 documentation - Boto3 Docs 1.17.7 documentation

On your local start python3, then in python environment work with Boto

import boto3

session = boto3.Session()

// starts a session

….

s3 = session.resource(‘s3’)

s3_client = session.client(‘s3’)

Working with files with Boto

import os

os.listdir()

os.rename(“file1.jpg”, “file2.jpg”)

Create s3 bucket and upload file to s3

Documentation:

S3 - Boto3 Docs 1.17.7 documentation

S3 - Boto3 Docs 1.17.7 documentation

S3 - Boto3 Docs 1.17.7 documentation

make a boto3 client

client = boto3.client(“s3”)

make a bucket in your local terminal boto3 client

response = s3_client.create_bucket(Bucket=”bucketname”,CreateBucketConfiguration={“LocationConstraint”:”us-east-2"})

check by printing response

print(response)

{‘ResponseMetadata’: {‘RequestId’: ‘A0F81FDFDF3FD2C1’, ‘HostId’: ‘kaCye244upz6ZD3/L2rQ456D3vdkJys9w+rT4566645s5ZAN56ZdUxu71kOXw+wk/OoShiU=’, ‘HTTPStatusCode’: 200, ‘HTTPHeaders’: {‘x-amz-id-2’: ‘kENCye4Xpfupz6ZD3/L2rQdARVD3vdkJys9w+rTPQ8kewPx0xZAN56ZdUxu71kOXw+wk/OsShiU=’, ‘x-amz-request-id’: ‘A0ASDFDFDFDSFD2C1’, ‘date’: ‘Tue, 16 Feb 2021 03:01:57 GMT’, ‘location’: ‘http://bucketname.s3.amazonaws.com/', ‘content-length’: ‘0’, ‘server’: ‘AmazonS3’}, ‘RetryAttempts’: 0}, ‘Location’: ‘http://bucketname.s3.amazonaws.com/'}

List all buckets in s3

for key in s3_client.list_buckets()[‘Buckets’]: print(key[‘Name’])

upload a file from local to s3 bucket

optional 1- convert the file as binary

with open(“filename.png”,”rb”) as f:

data=f.read()

…

now put the file using this

S3 - Boto3 Docs 1.17.7 documentation

response = client.put_object(ACL =”private”, Body=data, Bucket=”filename.png”, Key =”filename.png”,)

*** key same as filename always

____

option2

s3_client.upload_file(‘filename.png’, ‘bucketname’, ‘filename_v2.png’)

___

option3

s3.Bucket(‘bucketname’).Object(‘filename2.png’).upload_file(‘filename_local.png’)

or in folder

s3.Bucket(‘bucketname’).Object(‘foldername/filename2.png’).upload_file(‘filename_local.png’)

List the files

for obj in bucket.objects.all():

print(obj.key)

Delete a file in S3

Option1-

response = client.delete_object(Bucket =”bucketname”, Key=”filename.png”)

Option 2-

bucket=s3.Bucket(“bucketname”)

response = bucket.delete_objects(Delete ={“Objects”:[{“Key”:“filename.png”},],},)

Option 3-

bucket = s3.Bucket(‘bucketname’) bucket.objects.all().delete() bucket.delete()

S3 - Boto3 Docs 1.17.7 documentation

List all files in s3 bucket

S3 - Boto3 Docs 1.17.7 documentation

Option1

bucket = s3.Bucket(‘bucketname’)

for obj in bucket.objects.all(): print(obj.key)

…

option2

response = client.list_objects(
    Bucket='string',
   
)

response = client.list_objects(Bucket=”bucketname”,)

response

will give you a big jason output … look for contents

‘Contents’: [{‘Key’: ‘filename.png’,}]

or write a for loop

for x in response.get(“Contents”,None):print(x.get(“Key”,None))

List all Buckets

option1

response = client.list_buckets()

Response

this will return a big jason reply… to get the buckets name

for x in response.get(“Buckets”,None): print(x.get(“Name”,None))

Option2

for key in s3_client.list_buckets()[“Buckets”]:print(key[“Name”])

Download from s3 to local:

s3_client.download_file(‘bucketname’, ‘filename.png’, ‘filename_on_local.png’)

Delete bucket

bucket = s3.Bucket(‘bucketname’)

bucket.delete()

Delete all items in bucket

you have to delete all items in bucket before the bucket

bucket = s3.Bucket(‘bucketname’)

bucket.objects.all().delete()

bucket.delete()

Create a folder ABC and put object in that folder:

response = s3_client.put_object( Bucket=’bucketname’, Key=(‘ABC/filename2.png’))

print(response)

Copy a file from one bucket to another

copy_source = { ‘Bucket’: ‘bucketname1’, ‘Key’: ‘filename1.png’ }

s3_client.copy(copy_source, ‘bucketname2’, ‘filename2.png’)

from between folders:

copy_source = { ‘Bucket’: ‘bucketname1’, ‘Key’: ‘foldername/filename1.png’ }

s3_client.copy(copy_source, ‘bucketname2’, ‘foldername/filename2.png’)

Other items:

S3 - Boto3 Docs 1.17.7 documentation

• Bucket.copy()
• Bucket.create()
• Bucket.delete()
• Bucket.delete_objects()
• Bucket.download_file()
• Bucket.load()
• Bucket.put_object()
• Bucket.upload_file()
• Bucket.upload_fileobj()