Open edX and Email Automation

If you use Open edX to run your online courses, sooner or later you’ll realize that you need better emailing capability.

Open edX does give you a good start – sending out an email after a user registers, or forgets their password, or wants a digest of forum posts – but beyond that you’ll have to do a manual “Bulk” email via the Instructor panel when you want to, say, remind students that the course is ending in a week. If you’ve done transactional email in an ecommerce setting, you know there’s lots more you can do with email to keep users engaged.

The Open edX team does list a bunch of emails you should probably send out using the bulk email form, as a way of keeping users informed and engaged. But manual operations are such a drag!

For the Open edX site I’m building for my client, I want email automations to handle a lot of this manual work. Open edX has a great big list of events that get bounced around the system, so why not hook into those and set up some automationed emails to trigger when they occur. Yes, great idea! Hmmm…ehhh now how to do that….

Automate Emails for Some Important Events

To start I wanted an automated email to be sent for just four particular events:

• Registration : edx.bi.user.account.registered
• Enrolled in a course : edx.course.enrollment.activated
• Unenrolled from a course : edx.course.enrollment.deactivated
• Certificate earned : edx.certificate.created

Sending some nicely designed, personalized emails for these events would go a long way towards improving the student’s experience and keeping them engaged with the site.

To be fair, the Open edX system does send a standard registration email with a authorize link to complete the rego, and you could overload that to include lots of marketing stuff, but I wanted something much snazzier, plus extra tagging, plus adding to newsletters, plus other automations to be kicked off when somebody joins our community. I WANT MORE.

Automated Email SaaS

A key part in any kind of email automation system is automating the emails. For realz. And there are a ton of SaaS providers that do just that. Given the ferocious competition in this sector, they’re ultimately better, cheaper and more reliable than anything I would ever build myself. So going with an email automation SaaS was a no-brainer.

I scouted around a bit, compared prices, kicked the tires, and ended up going with ActiveCampaign as a transactional email provider. They had the most reasonable pricing plan, given my client is a non-profit, e-learning-focused project, not a moneybags gotta-spend-to-earn runway-be-damned ecommerce play.

ActiveCampaign has a really nice UI for setting up automations and a reasonable API, and it was quite easy to get it set up some workflows and template emails for my four events. The more pressing question was how to connect Open edX events to these automations.

A Mini Data Broker

Open edX comes with some baked-in code for sending events to Segment.io, which I could ostenisbly use to connect to ActiveCampaign. Segment.io is pretty nifty but can get awful expensive awful fast. In this situation, all I needed was a simple data broker to receive Open edX events and do a few things: pass on certain events to ActiveCampaign, maybe send a note to our team’s Slack channel, and maybe even send a custom event to Google Analytics.

But rather than rely on (and pay for!) Segment, I built a simple Amazon Lamda function to parse incoming Open edX events from our EC2 instance and pass them on to those different services I wanted to call (mainly ActiveCampaign) via their API. This ‘mini data broker’ as I call my ugly baby gets the job done.

So basically my system looks like this.

The Lambda function was relatively easy to write: parse the incoming Open edX event, decide which services to send pieces of that data to, call those APIs, and le voila.

To get the Open edX app to send the events, I created a fork of event-tracking. This Django app is already structured to allow you to create different types of backends to send events to. My for includes a simple backend for AWS Lambda. Very simple. Rather ugly. But it’s serviceable: awslambda.py

Now, I’m only really concerned with just a few event types – a student enrolled, a student created a certificate, and so on – out of a whole lot of event types generated by Open edX. So I set up a whitelist in edx-platform/lms/envs/common.py that lists the specific events I care about. That will filter out all the other events generated as students interact with the system :

 'aws_lambda': {
        'ENGINE': 'eventtracking.backends.routing.RoutingBackend',
        'OPTIONS': {
            'backends': {
                'awslambda': {
                    'ENGINE': 'eventtracking.backends.awslambda.AwsLambdaBackend'
                }
            },
            'processors': [
                {
                    'ENGINE': 'eventtracking.processors.whitelist.NameWhitelistProcessor',
                    'OPTIONS': {
                        'whitelist': ['edx.bi.user.account.registered',
                                      'edx.course.enrollment.activated',
                                      'edx.course.enrollment.deactivated',
                                      'edx.certificate.created',
                                      ]
                    }
                }
            ]
        }
    }

And, as you’ll noticed in the __init__ function, I’m using boto3 to communicate with AWS Lambda. So I’ve created some extra environmental variables in edxapp to hold the credentials for connecting with Lambda.

def __init__(self):
        """
        Connect to Lambda
        """
        self.lambda_arn = settings.AWS_EVENT_TRACKER_ARN
        access_key = settings.AWS_ACCESS_KEY_ID
        secret_key = settings.AWS_SECRET_ACCESS_KEY
        aws_region = getattr(settings, "AWS_EVENT_TRACKER_REGION", "us-west-1")

        self.client = boto3.client('lambda',
                                   aws_access_key_id=access_key,
                                   aws_secret_access_key=secret_key,
                                   region_name=aws_region)

To install your own fork of event-tracking, you need to make sure that you replace the Open edX event-tracking dependency with your own fork. I did this by moving the dependency to github.txt and setting it to my fork’s url. This might be heresy. I don’t know.

Notes

There’s a few small hurdles to jump when trying to send events this way.

• Some places in the edxapp code don’t send the event to the event-tracking tracker, but rather a different instance of this library included directly in the edxapp code. So I had to either switch this code to use event-tracking or create a separate call for my tracker.
• I had to update the Ansible build to include values for AWS_EVENT_TRACKER_ARN, AWS_EVENT_TRACKER_REGION (and maybe different environmental variables for the access key and secreate access key if you create a different IAM role for this particular connection).

I’ve got to stop at the moment but I’ll be back in the near future to fill this post out a little more.

Other Approaches

If you’ve drunk the AWS cool-aid and see everything in terms of their bagillion services, another way of doing this is to set up a CloudWatch Log Agent on your EC2 instance running Open edX, set it to watch /edx/var/logs/lms/edx.log and then create a filter and trigger to call your Lambda function whenever you see an Open edX event you care about, like edx.bi.user.account.registered.

This approach gets rid of the need to fork event-tracking, but does couple your system more to AWS.

Annnnnd I’m probably missing lots of other ways of doing this. Send me a note if you have any thoughts and I’ll update this post…or just catch me in one of the Open edX Slack channels, where I frequently ask silly questions.