Recent Posts

#20 Install aws amazon sdk on Raspberry Pi 3 and Test connection (Publish - Subscribe IoT Architecture)

IoT Tutorial #20 [ Raspberry Pi Tutorials #8 ]
In this post,
We will send messages over the internet from one python script (aws_iot_pub.py) to another python script (aws_iot_sub.py) running on Raspberry Pi 3 using aws amazon cloud (publish-subscribe IoT Architecture)

In order to use Amazon cloud services on Raspberry Pi, we need to install aws amazon sdk on Raspberry Pi (Raspbian stretch OS).

Steps to install aws amazon sdk om Raspberry Pi 3 and Test connection:

1.       Sign up on AWS Amazon website and (login from Raspberry Pi)
 and Go to: Services > AWS IoT (IoT Core) > Onboard > Get Started
a.       Choose Platform: Linux/OSX
b.      Choose an AWS IoT Device SDK: Python
c.       Set any name of a thing (we will refer it as <Thing_Name>)
d.      Download connect kit on Raspberry Pi.
 Click on Next > Next > Finish/Done to complete the configuration of the device/thing.

2.       Install:
  Unzip the downloaded file. You should have following files:
a.       <Thing_Name>.cert.pem
b.      <Thing_Name>.private.key
c.       <Thing_Name>.public.key
d.      start.sh
                    Put all the unzipped files in one folder.




3.       Run following commands to Check Versions and install paho-mqtt:
$ openssl version
OpenSSL 1.1.0f  25 May 2017
$ python –V
Python 2.7.13
$ openssl version
OpenSSL 1.1.0f  25 May 2017
$ sudo pip install paho-mqtt
$ sudo  apt-get update

4.       Download "root-CA.crt" and sample sdk using start.sh script:
  Give executable permissions to "start.sh"

 Open terminal go to above unzipped directory
 Run following commands:
$ pip install AWSIoTPythonSDK
$ ./start.sh
And wait for the installation to complete...
Try it 2-3 times, It might show an error similar to:
error: could not create '/usr/local/lib/python2.7/dist-packages/AWSIoTPythonSDK': Permission denied
But try again and again 2-3 times till you see msg similar to below on terminal and press "ctrl + c"
{"message": "Hello World!", "sequence": 5}
  "root-CA.crt" & sample SDK is now downloaded in the same folder as that of "start.sh"

NOTE: If "root-CA.crt" is not downloaded, then click here to download. 


Steps to test the connection with aws cloud:

5.       Download "aws_iot_pub.py" and "aws_iot_sub.py" and
 paste it to the folder having all aws unzipped files

 Make the changes in both python script (as mentioned in those python script)
 Provide Executable permission to both python script.
 Run both python scripts on RPi3.

You will see all the messages published by "aws_iot_pub.py" received by "aws_iot_sub.py"





NOTE:
·         aws_iot_pub.py will generate random temperature data in the range of 20 to 25 degree celcius and send it to the amazon aws broker to the topic “temperature
·         aws_iot_sub.py will receive all the messages published to all the topics on registered thing on amazon aws cloud.


  THESE MEANS YOUR ARE CONNECTED TO "AWS AMAZON IOT BROKER" SUCCESSFULLY!!!

For Better understanding you can watch demonstration video given below:

Downloads:

Download link is given in the Description of the YouTube video shown below.


Demonstration:




aws_iot_pub.py:

#!/usr/bin/python

############################################ 
# PROBLEM STATEMENT:
# This program will publish test mqtt messages using the AWS IOT hub
# 
# To test this program you have to run first its companinon aws_iot_sub.py
# that will subscribe and show all the messages sent by this program
#
############################################

############################################
# STEPS:
#
# 1. Sign in to AWS Amazon > Services > AWS IoT > Settings > copy Endpoint
#    This is your awshost
# 
# 2. Change following things in the below program:
#    a. awshost   (from step 1)
#    b. clientId  (Thing_Name)
#    c. thingName (Thing_Name)
#    d. caPath    (root-CA_certificate_Name)
#    e. certPath  (<Thing_Name>.cert.pem)
#    f. keyPath   (<Thing_Name>.private.key)
# 
# 3. Paste aws_iot_pub.py & aws_iot_sub.py python scripts in folder where all unzipped aws files are kept. 
# 4. Provide Executable permition for both the python scripts.
# 5. Run aws_iot_sub.py script
# 6. Run this aws_iot_pub.py python script
#
############################################

# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
from time import sleep
from random import uniform
 
connflag = False
 
def on_connect(client, userdata, flags, rc):                # func for making connection
    global connflag
    print "Connected to AWS"
    connflag = True
    print("Connection returned result: " + str(rc) )
 
def on_message(client, userdata, msg):                      # Func for Sending msg
    print(msg.topic+" "+str(msg.payload))
 
#def on_log(client, userdata, level, buf):
#    print(msg.topic+" "+str(msg.payload))
 
mqttc = paho.Client()                                       # mqttc object
mqttc.on_connect = on_connect                               # assign on_connect func
mqttc.on_message = on_message                               # assign on_message func
#mqttc.on_log = on_log

#### Change following parameters #### 
awshost = "a1r7yoit83fln7.iot.us-east-2.amazonaws.com"      # Endpoint
awsport = 8883                                              # Port no.   
clientId = "aws_thing1"                                     # Thing_Name
thingName = "aws_thing1"                                    # Thing_Name
caPath = "root-CA.crt"                                      # Root_CA_Certificate_Name
certPath = "aws_thing1.cert.pem"                            # <Thing_Name>.cert.pem
keyPath = "aws_thing1.private.key"                          # <Thing_Name>.private.key
 
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)  # pass parameters
 
mqttc.connect(awshost, awsport, keepalive=60)               # connect to aws server
 
mqttc.loop_start()                                          # Start the loop
 
while 1==1:
    sleep(5)
    if connflag == True:
        tempreading = uniform(20.0,25.0)                        # Generating Temperature Readings 
        mqttc.publish("temperature", tempreading, qos=1)        # topic: temperature # Publishing Temperature values
        print("msg sent: temperature " + "%.2f" % tempreading ) # Print sent temperature msg on console
    else:
        print("waiting for connection...")                  





aws_iot_sub.py:

#!/usr/bin/python

############################################ 
# PROBLEM STATEMENT:
#
# This program will subscribe and show all the messages sent by its companion 
# aws_iot_pub.py using AWS IoT hub
#
############################################

############################################
# STEPS:
#
# 1. Sign in to AWS Amazon > Services > AWS IoT > Settings > copy Endpoint
#    This is your awshost
# 
# 2. Change following things in the below program:
#    a. awshost   (from step 1)
#    b. clientId  (Thing_Name)
#    c. thingName (Thing_Name)
#    d. caPath    (root-CA_certificate_Name)
#    e. certPath  (<Thing_Name>.cert.pem)
#    f. keyPath   (<Thing_Name>.private.key)
# 
# 3. Paste aws_iot_pub.py & aws_iot_sub.py python scripts in folder where all unzipped aws files are kept. 
# 4. Provide Executable permition for both the python scripts.
# 5. Run aws_iot_sub.py script
# 6. Run this aws_iot_pub.py python script
#
############################################

# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
 
def on_connect(client, userdata, flags, rc):                # func for making connection
    print("Connection returned result: " + str(rc) )
    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("#" , 1 )                              # Subscribe to all topics
 
def on_message(client, userdata, msg):                      # Func for receiving msgs
    print("topic: "+msg.topic)
    print("payload: "+str(msg.payload))
 
#def on_log(client, userdata, level, msg):
#    print(msg.topic+" "+str(msg.payload))
 
mqttc = paho.Client()                                       # mqttc object
mqttc.on_connect = on_connect                               # assign on_connect func
mqttc.on_message = on_message                               # assign on_message func
#mqttc.on_log = on_log

#### Change following parameters ####  
awshost = "a1r7yoit83fln7.iot.us-east-2.amazonaws.com"      # Endpoint
awsport = 8883                                              # Port no.   
clientId = "aws_thing1"                                     # Thing_Name
thingName = "aws_thing1"                                    # Thing_Name
caPath = "root-CA.crt"                                      # Root_CA_Certificate_Name
certPath = "aws_thing1.cert.pem"                            # <Thing_Name>.cert.pem
keyPath = "aws_thing1.private.key"                          # <Thing_Name>.private.key
 
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)      # pass parameters
 
mqttc.connect(awshost, awsport, keepalive=60)               # connect to aws server
 
mqttc.loop_forever()                                        # Start receiving in loop

UPDATES:

Many of you were facing the issue like : "Waiting for connection...

The reason behind it is, In a year, few things are changed on AWS, one of them is policy associated with the thing we create.
As per the new policies, After successful download and installation of the "root-CA.crt" & Sample SDK (As shown in step 4), A sample Pub-sub program run on our Raspberry Pi.

Now if you go to policies associated with our newly created thing on aws, you will see the parameter "Resource" and its value is set to some python or java program (These scripts are nothing but the default scripts from SDK to run a sample pub-sub). So that, only those scripts can access our aws thing.


Now, In order to run our pub-sub scripts you have to do either of the change:

  • Either Add your python script name in the Resource list in the policy
  • or Remove all the name from Resource parameter and replace it by "*"
NOTE: "*" means Set Resource as all.

The good news is, Solution is available for it. Detailed steps are as given below:

Follow the below steps:
  1. Login to your AWS account, Go to AWS IOT (IOT Core)
  2. Go to Secure > Policies > Click on the Policy Associated with your thing.
    Eg.If your thing name is aws_thing1 then the policy related to this may be aws_thing1-Policy
  3. Click on Edit policy document
  4. Delete everything and paste following in that space:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "iot:Publish",
            "iot:Subscribe",
            "iot:Connect",
            "iot:Receive"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
  5. Click on save as a new version
  6. Problem is solved. Now run your aws_iot_pub.py and aws_iot_sub.py on your Raspberry Pi.
--------------------------------------------------------------------------------

Click here to see more codes for Raspberry Pi 3 and similar Family.
&
Click here to see more codes for NodeMCU ESP8266 and similar Family.
&
Click here to see more codes for Arduino Mega (ATMega 2560) and similar Family.

Feel free to ask doubts in the comment section. I will try my best to solve it.
If you find this helpful by any mean like, comment and share the post.
This is the simplest way to encourage me to keep doing such work.

Thanks and Regards,
-Akshay P. Daga

33 comments:

  1. Nice Post.
    In this article the publisher and subscriber on the same client (Raspberry PI), what if the publisher is outside somewhere from the cloud (AWS). What are steps to turning Lambda/ SNS/ etc as a publisher to this AWS IOT Mqtt broker?

    ReplyDelete
  2. Hi, do you have any idea how to solve this?
    pi@raspberrypi:~/downloadedaws $ ./start.sh
    bash: ./start.sh: Permission denied

    Thanks in advance

    ReplyDelete
    Replies
    1. https://askubuntu.com/questions/409025/permission-denied-when-running-sh-scripts
      Got it

      Delete
  3. Hi, ran into another problem, I have done everything as instructed but I am getting "waiting for connection..." only as the output. But my .sh file ran successfully and i was able to see the "hello" messages on AWS.
    Thanks in Advance

    ReplyDelete
  4. i'm also having save message "waiting for connection ...."

    ReplyDelete
    Replies
    1. Thank for your patience. I found the solution and updated the blog.
      Check out the UPDATES Section above.

      If you like my work, Please subscribe to my YT channel: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1
      It will be a great help for me.
      Thanks

      Delete
  5. i am getting waiting for connection problem

    ReplyDelete
    Replies
    1. @Akash shrimali, Thank for your patience. I found the solution and updated the blog.
      Check out the UPDATES Section above.

      If you like my work, Please subscribe to my channel: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1
      It will be a great help for me.
      Thanks

      Delete
  6. i am getting waiting for connection problem

    ReplyDelete
    Replies
    1. My Amazon AWS account is deactived now. So, I am not able to try it out to answer your query.

      Please open your own amazon aws account. Make above mentioned changes in all the codes.
      Above program should try to connect to your aws account. And it will work.

      Delete
    2. Change following things in the below program:
      # a. awshost (from step 1)
      # b. clientId (Thing_Name)
      # c. thingName (Thing_Name)
      # d. caPath (root-CA_certificate_Name)
      # e. certPath (.cert.pem)
      # f. keyPath (.private.key)

      and try again.
      You need to debugging using "print" statement to check upto which line it is working fine.

      Because when I tried, It was working as shown in the video. I have uploaded the same working code from the video without editing anything.

      Please check & let us know if you get something.
      Thanks

      Delete
    3. Connflag is not set to true. that's the problem I'm having

      Delete
    4. In both files on_connect func is not executed.

      Delete
    5. Does it have something to do with on_connect function, because i dont see that function used anywhere in the program.

      Delete
    6. on_connect gets automatically called internally by mqttc.loop_start(). We don't have to call it explicitly.

      Delete
    7. @SJ SONI, @Unknown, @Akash Shrimali,
      I have tried it today from the scratch. I copied the above 2 codes and followed complete steps shown in above video.
      and My connection was successful. My randomly generated temperature data was sent from publish script to subscribe console through aws cloud using MQTT protocol.

      There must be something wrong or missing while configuring, aws account. Above given codes are correct for sure. Tested it once again.

      Delete
    8. Since many of you are getting this error. I am debugging it.
      Will let you know once found the solution.

      Delete
    9. @Akash shrimali, @SJ SONI Thank for your patience. I found the solution and updated the blog.
      Check out the UPDATES Section above.

      If you like my work, Please subscribe to my YT channel: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1
      It will be a great help for me.
      Thanks

      Delete
  7. Actually Amazon has updated the Root CA certificate. So you have to use the new one
    Link : https://docs.aws.amazon.com/iot/latest/developerguide/managing-device-certs.html
    This is how i solved the problem of not getting connection?

    ReplyDelete
    Replies
    1. Which one do you use? I tried all and they did not work but the first one showed a certificate verify failed instead of waiting for connection?

      Delete
    2. @Knowledge Shelf, Thanks for your Input.
      I have gone through your videos and blogs. Really nice work. Keep doing it.
      We should collaborate some time.

      I tried all the certificates from the link you have provided. Nothing worked for me.
      VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem > ERROR
      AmazonRootCA1.pem > Waiting for connection
      AmazonRootCA2.pem > ERROR
      AmazonRootCA3.pem > ERROR
      AmazonRootCA4.pem > ERROR

      Can you please suggest what exactly needs to be done?

      Delete
  8. Dear Akshay,
    Thanks for your sharing. Your post let me learn a lot of knowledge about AWS. Thank you.

    ReplyDelete
    Replies
    1. Thank you. Glad to hear that.
      Please subscribe to the channel here: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1
      It will be great help for me.

      Delete
  9. Hi, the video has been a big help but in running the program I am getting an error saying
    Traceback (most recent call last):
    File "/home/pi/connectionL/aws_iot_sub.py", line 65, in
    mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None) # pass parameters
    File "/home/pi/.local/lib/python3.5/site-packages/paho/mqtt/client.py", line 764, in tls_set
    context.load_cert_chain(certfile, keyfile)
    FileNotFoundError: [Errno 2] No such file or directory
    Any help would be appreciated, Thanks!

    ReplyDelete
    Replies
    1. Actually I am also getting the same error now. Some changes have been made by AWS server over the period. I am working on it. I will let you know the solution for that error.

      Meanwhile, Please try to download the Root CA certificate from the link given by "Knowledge Shelf" in comments section.
      It might work.

      Delete
    2. That didn't work. Let me know when you figure out a fix, Thanks I appreciate it

      Delete
    3. Hi Alex Louis,
      Thank for your patience. I found the solution and updated the blog.
      Check out the UPDATES Section above.

      If you like my work, Please subscribe to my YT channel: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1
      It will be a great help for me.
      Thanks

      Delete
    4. Thank you, it works. Will this policy work for any python code using Paho or is something specific to your code?

      Delete
    5. Yes. It will work for any python code.

      Delete
  10. Hello, I have followed all the above instructions properly and it is working correctly. Thank you.
    But I am using ultrasonic sensor for my any other project. I have made the changes in pub.py file it publishing correctly. But the output is not seen on sub.py file. How can I solve it?

    ReplyDelete
    Replies
    1. Glad to know that it works for you.
      You have to debug it line by line.
      and Make sure you sub.py subscribes to the same topic for which your pub.py is publishing.

      If you find my blog helpful, Please subscribe to my YT channel: https://www.youtube.com/c/APDagasDumpBox?sub_confirmation=1

      Delete
  11. I am having a ssl issue.I get this "ssl.SSLError: [SSL] PEM lib (_ssl.c:2699)"

    ReplyDelete
    Replies
    1. Did you check the "UPDATES:" section in the bottom of the post. That might solve your problem.

      Delete