≡ Menu

How to build a serverless SMS Weather Bot using Twilio and AWS

As an avid fisherman on Lake Champlain, I’m constantly checking the National Weather Service’s lake forecast.  This tells me three key pieces of information:

  1. How windy is it?
  2. Which direction is the wind blowing in?
  3. How big are the waves?

These three pieces of information will determine where I’ll launch the boat and where I’ll fish.  And in some cases whether I’ll fish at all.  Fishing in 3-4 foot waves can be pretty miserable.

I’ve recently been using the Twilio SMS API to handle messaging for work and I noticed they had some newer workflow options to build SMS and voice flows, so I thought it’d be interesting to build a simple SMS weather bot.  To do this, I knew I’d need to tie a few different pieces together.  Here they are in order:

  1.  A way to scrape the forecast page every morning when the new forecast is published and extract just the text I needed.  For this I used an AWS Lambda function with a Cloudwatch Event running on a CRON schedule.
  2. An easy way to store the scraped forecast data so I don’t have to scrape it more than once per day.  DynamoDB fit the bill here.
  3. A way to serve up the stored forecast data. API Gateway + Lambda.
  4. An SMS # to receive and send messages.  Twilio.
  5. A workflow to build logic against the SMS # so I can call the API that serves up the forecast data.

Note: The lake forecast has recently expanded to include 5 days of data.  To keep things simple for this demo, I’m only using three forecast periods: “today”, “tonight” and “tomorrow”.

You can out my Lake Weather SMS bot by sending any message to: +1 202-999-3555

AWS + Twilio Lake Weather Forecast Application

Here’s a diagram showing the overall process and how everything fits together.

Note: You can build this entire application for free

Please note that free trials are available for both Twilio and AWS which will allow you to build this entire application free of charge.  You can read more about the AWS Trial and the Twilio Trial on their respective web sites.

Let’s get building!

The Lake Champlain Forecast page

Let’s take a look at the page containing the forecast we’ll be parsing (https://forecast.weather.gov/product.php?site=BTV&issuedby=BTV&product=REC&format=txt&version=1&glossary=0).  If you view source, you’ll see a preformatted <pre> tag (<pre class=”glossaryProduct“>) that contains the text we care about:

<pre class="glossaryProduct">
000
SXUS41 KBTV 070656
RECBTV
NYZ028>031-034-035-VTZ001>012-016>019-072115-

Recreational Forecast
National Weather Service Burlington VT
256 AM EDT Fri Sep 7 2018

.The Lake Champlain Forecast...

.TODAY...North winds 5 to 10 knots. Waves around 1 foot.
.TONIGHT...North winds 5 to 10 knots, becoming northwest 10 to
20 knots after midnight. Waves around 1 foot, building to 1 to
2 feet after midnight.
.SATURDAY...North winds 10 to 20 knots, becoming 10 to 15 knots in
the afternoon. Waves 1 to 2 feet.
.SATURDAY NIGHT...Northeast winds 10 to 15 knots, becoming north
5 to 10 knots after midnight. Waves 1 to 2 feet.
.SUNDAY...North winds around 5 knots, becoming northeast in the
afternoon. Waves 1 foot or less, subsiding to around 1 foot in the
afternoon.

The parts we care about are TODAY, TONIGHT and SATURDAY (tomorrow).  If we look closely we’ll see that each piece we care about is loosely separated by three dots (…).  If we process the text contained within the <pre> tag as a string and split it into an array on the three dots, we’ll end up with:

First array item (position 0 in JavaScript):

000
SXUS41 KBTV 070656
RECBTV
NYZ028>031-034-035-VTZ001>012-016>019-072115-

Recreational Forecast
National Weather Service Burlington VT
256 AM EDT Fri Sep 7 2018

.The Lake Champlain Forecast

Second array item (position 1):

.TODAY

Third array item (position 2) – Today’s forecast:

North winds 5 to 10 knots. Waves around 1 foot.
.TONIGHT

Fourth array item (position 3) – Tonight’s forecast:

North winds 5 to 10 knots, becoming northwest 10 to
20 knots after midnight. Waves around 1 foot, building to 1 to
2 feet after midnight.
.SATURDAY

This gives us enough to get started with.  We know we’ll want most of the text from the 3rd 4th and 5th (not shown) items in the array to get forecasts for ‘today’, ‘tonight’ and ‘tomorrow’.  We’ll also need to deal with the following day’s forecast title (.TONIGHT, .SATURDAY, etc.) which comes along for the ride.  If we look carefully, we’ll see that we can use Regex to split this string again since it’s a period followed by no space and then a upper case letter.  This results in the following RegEx in JavaScript: (/\r?\.[A-Z]/).  Once we split the string using this RegEx, we’ll take the first item in the array as our forecast.

The Lambda Web Scraper

To build a scraper in Lambda, we’re going to use NodeJS and load in a few libraries:

npm install --save request-promise cheerio aws-sdk
  1. request-promise – Allows us to make a web request to the NOAA forecast page.
  2. cheerio – Parses the HTML easily.
  3. aws-sdk – Includes the DynamoDB SDK.

Here’s the code of index.js in its entirety:

const rp = require('request-promise');
const cheerio = require('cheerio');
var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10', region: 'us-east-1'});

exports.handler = (event, context, callback) => {

    // Set the options for our Scrape.  The URI to scrape and a User Agent so we don't get blocked.
    const options = {
        uri: 'https://forecast.weather.gov/product.php?site=BTV&issuedby=BTV&product=REC&format=txt&version=1&glossary=0',
        headers: {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36'
        },
        transform: function (body) {
          // Load the retrieved page into Cheerio so we can parse it
          return cheerio.load(body);
        }
      };

  // Get the weather page and parse it 
  rp(options)
  .then(($) => {
    let rawForecast = $('pre').html();
    // Split the weather data on '...'
    let splitForecast = rawForecast.split("...");

    // Dynamo DB params
    var params = {
        ExpressionAttributeNames: {
         "#F": "Forecast"
        }, 
        ExpressionAttributeValues: {
         ":f": {
           S: splitForecast[2].split(/\r?\.[A-Z]/)[0] // Splits the weather data on .[A-Z] so we don't get the next day's prefix
          }
        }, 
        Key: {
         "forecastperiod": {
           S: "today"
          }
        }, 
        ReturnValues: "ALL_NEW", 
        TableName: "LakeForecast", 
        UpdateExpression: "SET #F = :f"
       };
       dynamodb.updateItem(params, function(err, data) {
         if (err) console.log(err, err.stack); // an error occurred
         else {
                //Update tonight
                params.ExpressionAttributeValues[":f"].S = splitForecast[3].split(/\r?\.[A-Z]/)[0];
                params.Key.forecastperiod.S = "tonight";
                dynamodb.updateItem(params, function(err, data) {
                if (err) console.log(err, err.stack); // an error occurred
                else {
                     //Update tomorrow
                    params.ExpressionAttributeValues[":f"].S = splitForecast[4].split(/\r?\.[A-Z]/)[0];
                    params.Key.forecastperiod.S = "tomorrow";
                    dynamodb.updateItem(params, function(err, data) {
                        if (err) console.log(err, err.stack); // an error occurred
                        else {
                            // Let Lambda know we're all done processing.
                            callback(null, 'all done processing!');
                        }        
                      });
       
                }        
              });
         }        
       });
  })
  .catch((err) => {
    callback(err);
  });
};

There’s definitely some room for improvement with the text parsing (splitting on ‘…’, etc.) but this will do for now.  Once we’ve created our index.js we’ll need to zip up the files to include our node_modules folder which contains the libraries and their dependencies.

Our lambda function will need to be assigned an IAM role that allows it to execute the Lambda function (Lambda basic execution) and write data to DynamoDB.  If you aren’t familiar with IAM, let me know in the comments and I’m happy to help.

Note that when you call the DynamoDB ‘updateItem’ method if a record doesn’t already exist with the key you provide, it’s created/inserted.  It’s really an ‘upsert’ operation.  This is handy for our needs so we don’t need to get the item first to see if it exists.  We can just call ‘updateItem’ and let DynamoDB figure it out behind the scenes.

Cloudwatch Events

To run this Lambda daily, we’re going to setup a cloudwatch event to run every day at 09:00 (9 AM) UTC which is after the Lake Champlain forecast is published each morning around 3 AM EDT/2 AM EST.  To do this, choose ‘Cloudwatch Events’ as a trigger at the top of the Lambda page and then scroll down to customize the Cloudwatch Event.  I entered a Cron expression of:

cron(0 9 * * ? *)

You can read more about the Cron expressions accepted by Lambda in this AWS doc.

Lambda Cloudwatch Event Trigger

Cloudwatch Event

DynamoDB

To hold our forecast data, we’ll just create a DynamoDB table, ‘LakeForecast’, with a key of ‘forecastperiod’ and we’ll pass in an attribute of ‘forecast’ when we load data.  You won’t need a secondary index since we’ll always be retrieving data using the exact key (e.g., today, tonight, tomorrow).  Here’s how our DynamoDB table will look once we’ve scraped and populated some records:

DynamoDB Forecast Table

API Gateway + Lambda

To retrieve the data from the Twilio flow using an API request, we need to setup an API Gateway with a POST method at the root.  When we want to retrieve our forecast, we’ll POST a JSON message specifying the forecast period we want to retrieve:

{
    "forecastperiod" : "today"
}

This JSON body will travel all the way through to our Lambda method where it’ll appear as the ‘event’ variable.  This means you’ll need to add a model under ‘Request Body’ in the Method Request with a content type of ‘application/json’.  In the Integration Request, we’ll allow the Request body passthrough as seen below.

API Gateway POST Method

 

API Gateway Method Request

API Gateway Integration Request Mapping Template:

API Gateway Mapping Template

 

Our API Gateway POST method will be integrated with a new lambda function, ‘lambda-weather-get-forecast’, we’ll create to retrieve the appropriate forecast data from DynamoDB:

API Gateway Integration Request

Lambda-weather-get-forecast index.js:

var AWS = require('aws-sdk');
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10', region: 'us-east-1'});

exports.handler = (event, context, callback) => {
    console.log(event);
    
    var params = {
        Key: {
         "forecastperiod": {
           S: event.forecastperiod.toLowerCase()
          }
        }, 
        TableName: "LakeForecast"
       };
       dynamodb.getItem(params, function(err, data) {
         if (err) callback(err); // an error occurred
         else {
             let reply = { message: data.Item.Forecast.S };
             callback(null, reply );          // successful response
         }
       });
}

When this Lambda function executes, it will return data like:

{ 
   "message": "North winds 5 to 10 knots. Waves around 1 foot."
}

This Lambda function will need an IAM role that can execute Lambda and read from DynamoDB.

Note that I originally created a resource with a path parameter of {forecastperiod} planning to use a GET request to /today, /tonight, /tomorrow, but unfortunately the Twilio widget for API requests doesn’t support a GET method with a response type of application/json only URL encoded forms.  Odd, right?

Twilio SMS #

Creating a new SMS # within Twilio is a really straight forward process.  You can do so on the Twilio create number page.  If you’re using the free trial, keep in mind that you’ll only be able to receive messages from Twilio on cell phones that you’ve verified.

Once you’ve created your Twilio number, go to this page and click on the number you’ve just created.  You’ll see a page like this:

This page is where you can integrate your Twilio number with webhooks, functions, etc.  In this case, I’ve specified that when a new message arrives, I want to kick off a ‘Studio Flow’ called ‘LakeWeatherSMS’.  This is where we’ll continue in the next and final step.

Twilio Flow

Twilio Flow gives us the ability to build workflows that are triggered from an SMS message, a phone call or a REST API call.  Flow uses Twilio Studio which means a large part of the workflow is built using a visual editor which is really nice since it illustrates the flow from each widget based on criteria such as success/fail, match/no match, etc.  Let’s take a look at the ‘LakeWeatherSMS’ flow at a high level then we’ll break down each piece (widget) individually.

Twilio LakeWeatherSMS Flow

Starting at the top of the flow, we have our trigger which fires on an incoming message.  This trigger was enabled in the previous step when we specified ‘LakeWeatherSMS’ as the flow to trigger when a new message arrives.  The arrow from the Trigger to the ‘sanitizeText’ widget indicates the execution direction and hints that information about the trigger is available to the sanitizeText function widget.

Twilio functions are serverless functions (FaaS), much like Lambda, that can be invoked via an API call or integrated into a Twilio flow.

Twilio Trigger

 

 

 

In the sanitizeText function I’ve specified a parameter ‘msgText’ which is mapped to the variable {{trigger.message.Body}} which as you might expect is the body of the SMS message received.  This parameter is passed to the sanitizeText function as the event which looks like:

{
   msgText: 'Today'
}

The sanitizeText function code is as follows:

exports.handler = function(context, event, callback) {
    console.log(event);
    
    var response = {
        cleanText: event.msgText.toLowerCase()
    }

	callback(null, response);
};

which returns:

{
   cleanText: 'today'
}

Yep, that’s it.  All it does is return a JSON object with one field, ‘cleanText’, that contains the event message text set to lower case.  This is helpful as most phones automatically capitalize the first letter of a text message (today becomes Today, etc.).  To simplify the matching case for the next widget, I want it set to lowercase.  I also wanted to see how Twilio functions worked for future applications.

Twilio Flow Section 2

In the next section, the execution path splits based on the message body we received from our Twilio function.  If the variable {{widgets.sanitizeText.parsed.cleanText}} (returned by the sanitizeText function) matches any of ‘today’, ‘tonight’, or ‘tomorrow’, then the execution path continues down the right side to call the API.  If it doesn’t, we continue down the left side and send a message, “Sorry, we couldn’t understand your request. Please reply with which forecast you would like: today, tonight or tomorrow.”  One important thing to note here is that the ‘send_instructions’ widget does not have any additional steps.  This means that the flow execution ends and a new incoming message would trigger a new flow execution.

The ‘split_incoming’ widget looks like this:

Twilio Split Based On Message Config

Twilio Split Based On Message Transitions

On the left side, we just name the widget and which variable we’re splitting on.  On the right side, we determine what our next widgets will be based on whether the text matches or not.  Pretty straightforward.

 

 

 

 

 

 

 

 

 

 

Now let’s crack open the send_instructions widget to see how messages are sent from Twilio:

Send Message Config

Send Message Transitions

Here we can see the Message Body as well as the Send Message From and Send Message To Variables.  The Send Message From variable, {{flow.channel.address}}, maps to our Twilio SMS #.  The Send Message To variable, {{contact.channel.address}}, maps to the original message address that triggered the flow.  As you can see there are no transitions specified which means execution will end once this Send Message widget is executed.

 

 

 

Let’s take a look at the right side of our execution now.

Twilio API Call and Send Message

In this section, we’re continuing because our SMS contained the words, ‘today’, ‘tonight’, or ‘tomorrow’.  We’ll take this text and call the API Gateway we created to retrieve our forecast and then send a message back which contains the forecast text.  Then the execution ends.

 

 

 

 

 

 

 

 

Twilio HTTP Request Config

Twilio HTTP Request Transitions

Let’s take a look under the hood in the HTTP Request widget.  We’ve specified our Request URL which points at the API Gateway we created earlier as well as a Content Type of ‘application/json’ and then the Request Body that we’re posting to our API.  Here you can see we’re referencing a Twilio variable, {{widgets.sanitizeText.parsed.cleanText}}, which is the lower case text we got back from our Twilio function – ‘today’, ‘tonight’ or ‘tomorrow’.

On the transitions pane, we decide which widget to execute next based on Success or Fail.  In this demo, I’ve only chosen to execute the ‘send_forecast’ widget next on success.   In a failure scenario, we could send a message to the user along the lines of, “Sorry, unable to retrieve your forecast, please try again later.”

 

 

Now it’s time to look at our final widget, send_forecast, which sends an SMS to the user with the weather forecast text we received back from our API Gateway which looks something like:

{ 
   "message": "North winds 5 to 10 knots. Waves around 1 foot."
}

Twilio Send Message Config

In the Message Body field of the config, we can reference this value as {{widgets.call_weather_api.parsed.message}} which parses the JSON for us and makes the ‘message’ property easily accessible.  As we did before, we’re specifying where the message should be sent from (our Twilio SMS #) and who it should go to (the SMS # that triggered this workflow execution).

 

 

 

 

 

 

 

 

 

See this workflow in action

You can test this workflow out yourself by sending a text message to +1 202-999-3555.  If you specify a message text of ‘today’, ‘tonight’ or ‘tomorrow’, you’ll get a reply with the weather forecast.  Otherwise you’ll receive instructions on how to request a forecast.

Our new Twilio bot in action

Conclusion

Wow!  We just built an SMS bot that can retrieve the Lake Champlain Weather forecast by integrating AWS and Twilio products.

This application is just scratching the surface of what can be built with Twilio’s Flow product.  Flow could be used to build things like an SMS bot to provide order status for online grocery delivery, airport delays, and so on.  By integrating Flow with AWS via an API call, we’re able to tap into the huge variety of managed services that AWS offers.  In this example, our application is entirely serverless since we’re using all managed services such as API Gateway, Lambda, and DynamoDB on the AWS side and everything in Twilio’s stack is serverless.

A favor to ask

Did you find this post helpful?  Did I leave something important out that would help you and others build this application?  Have suggestions on how I could improve this blog post?  Please let me know in the comments below.  Thanks!

Comments on this entry are closed.