Adventures with Qualtrics, part 1: Custom Web Services and Piped Text

To create a feature in a pilot study I was running in December, I took a dive into Qualtrics API and custom web service building. In the process, I discovered a couple of workarounds and little-documented properties of both. The key to integrating them: piped text.

Piped Text: The Qualtrics Variable

With piped text, you can insert any embedded data and any answer your subject gave into (almost) any Qualtrics context.

If this doesn’t excite you, it should.

Let me rephrase. Piped text references the content of variables you can set. It can do this in conditional validation, display logic and survey flow. (You can’t make it into a GOTO, but that might be a good thing.) The documentation undersells this; this Qualtrics blog article does it a little more justice.

For my purposes, the most important insight goes unmentioned: you can use piped text to pass data to an external web service. That way, you can use data from an in-progress session as input for arbitrarily complex logic implemented in a programming language of your choice.

The approach

How does this work? First, you identify the shortcode for an answer or embedded field. Then, you insert it into the URL, like so:

http://your.service.URL/${e://Field/Identifier}/${q://QID1783/ChoiceTextEntryVField>

This will substitute the value of Field and the answer to question QID1783 in time for the redirect.

Qualtrics can call an external service in two ways.

  1. End-of-survey redirect. Qualtrics simply passes the torch to your service, which wraps up the session for your participant.
  2. Web Service step in Survey Flow. Your service will pass results back to Qualtrics, and they’ll be available for as embedded data for the following Qualtrics questions in that session. (With the “Fire and Forget” setting, this can be asynchronous.)

The external service then passes the results back to Qualtrics.

What’s the pass-back format?

“Pass results back to Qualtrics” glides over a big issue: Qualtrics documentation does not provide a list of valid return formats. The documentation and the only StackOverflow answer I could find both mention RSS as the only example of an acceptable format. The random number generator everyone uses for MTurk compensation, however, has a much simpler outcome: random=7. That’s hopeful, but what if you want to pass multiple values back? Docs don’t say.

I decided to test this out on a dummy web service I wrote in Sinatra. It turns out that Qualtrics will take data from JSON, XML, and URI query element. (That’s ?a=b&c=d – I owe this insight to Andrew Long at the Behavioral Lab.) You can try this out for yourself — just put down https://salty-meadow-86558.herokuapp.com/ as your Web Service in Qualtrics.

Pulling the API in

My project required more data to the custom Web service than Piped Text could conveniently pass, which meant that I needed to tangle the API. For that, see part two.

Adventures with Qualtrics, part 1: Custom Web Services and Piped Text

Executing nested rules with dragonfly

Rule nesting makes context-free grammars very powerful. It allows for brevity while preserving complexity — and dragonfly, the unofficial Python extension to Dragon Professional Individual, seems to promise that functionality with RuleRef, which “allows a rule to include (i.e. reference) another rule”.

But using RuleRef is less obvious than it would appear. How do you actually refer to the rules? How do you execute the actions that are associated with the referenced rules? And how do you ensure that dragonfly does not complain about rule duplication if you do this multiple times?

I will proceed step-by-step, but if you want to jump ahead to the solution, you can read it on GitHub.

If you’re unfamiliar with Dragonfly, do read this introduction to basic Dragonfly concepts in the Caster documentation.

Step 1: Include the rule with RuleRef

Let’s start with a toy grammar. In this grammar, we will have two rules that are not exported: that is to say, you can’t invoke them directly. We’ll call them simply RuleA and RuleB. (I will refer to them as “subrules” from here on out.)

# Rules proper
class RuleA(MappingRule):
    exported = False
    mapping = {
        "add <n>": Text('RuleA %(n)s'),
    }
    extras = [
        IntegerRef("n", 1, 10),
    ]

class RuleB(MappingRule):
    exported = False
    mapping = {
        "bun <n>": Text("RuleB %(n)s") ,
    }
    extras = [
        IntegerRef("n", 1, 10),
    ]

We’ll call the top-level rule RuleMain and include Rules A and B in the extras.

class RuleMain(MappingRule):
    name = "rule_main"
    exported = True
    mapping = {
        "boo <rule_b> and <rule_a>": Text("Rule matched: B and A!"),
        "fair <rule_a> and <rule_b>": Text("Rule matched: A and B!"),
    }
    extras = [
        RuleRef(rule = RuleA(), name = "rule_a"),
        RuleRef(rule = RuleB(), name = "rule_b")
    ]

The name argument of RuleRef takes care of the correspondence between the spec and the subrule. To get recognized, you do actually have to match the subrule’s spec by saying e.g. “boo bun three add five”.

This only carries out the Text("Rule matched: ...") action defined in the MainRule, though. To actually execute the subrules, we’ll need to add the Function action.

Step 2: Use (and mass-produce) Function

Dragonfly’s Function allows arbitrary code execution. However, you can only pass in a function reference, to which Function passes the right extras (seemingly) automagically. The caster documentation gives a useful but incomplete example:

def my_fn(my_key):
  '''some custom logic here'''

class MyRule(MappingRule):
  mapping = {
    "press <my_key>":     Function(my_fn),
  }
  extras = [
    Choice("my_key", {
      "arch": "a",
      "brav": "b",
      "char": "c"
    })
  ]

When you say “press arch”, my_fn gets called with the value of the my_key extra. But what if the mapping contained a reference to another rule in another extra? Would that also be passed to my_fn? It turns out that Function actually passes keyword arguments. If you name the argument to my_fn the same as the name of your extra, then my_fn will be called with the value of that extra. You’re not limited to one extra, either: for example, if we added an extra called towel to MyRule.extras, then def my_fn(towel, my_key) would receive both.

(If you define my_fn with **kwargs, it will receive all extras in a dict, including the default _node, _rule, and _grammar. This does lose the order in which the subrules were invoked, so you can’t just pass a general function that invokes all rules unless you’re happy with them being invoked alphabetically / in an arbitrary order. That was my first approach:

def execute_rule(**kwargs): # NOTE: don't use
    defaultKeys = ['_grammar', '_rule', '_node']
    for propName, possibleAction in kwargs.iteritems():
        if propName in defaultKeys:
            continue
        if isinstance(possibleAction, ActionBase):
            possibleAction.execute()

In this case, Rule A will be executed before Rule B, no matter the optionality or the order of utterance, just because of the kwargs key order. I played around with exploring the default extras, but I haven’t managed to figure out how to extract the order from the actual utterance to reorder the subrules automagically; that might require a deeper dive into Dragonfly than I’m ready for.)

You could write executeRuleA(rule_a) to run rule_a.execute(), then add Function(executeRuleA) to be executed alongside Text when the rule is matched. Unless you want to do different things for different rules, though, it is easiest to define a factory for functions that simply execute whatever extras you specify:

from dragonfly import Function, ActionBase

def _executeRecursive(executable):
    if isinstance(executable, ActionBase):
        executable.execute()
    elif hasattr(executable, '__iter__'):
        for item in executable:
            _executeRecursive(item)
    else:
        print "Neither executable nor a list: ", executable

def execute_rule(*rule_names):
    def _exec_function(**kwargs):
        for name in rule_names:
            executable = kwargs.get(name)
            _executeRecursive(executable)

    return Function(_exec_function)

This way, if you want to execute rule B before rule A, you can add execute_rule(['rule_b', 'rule_a']) to the action. Equivalently, you could use execute_rule('rule_b') + execute_rule('rule_a'). (Since both factories return a Function, their output can be added with other dragonfly Action elements.)

Step 3: Reusing subrule references in other rules

Let’s say you want to reuse your subrules in another rule, like so:

# Note: This doesn't execute the sub-actions at all
class CompoundMain(CompoundRule):
    spec = "did (<rule_a1> and <rule_b1> | <rule_b1> and <rule_a1>)"
    exported = True
    extras = [
        RuleRef(rule = RuleB(), name = "rule_b1"),
        RuleRef(rule = RuleA(), name = "rule_a1"),
    ]

If you add this to your grammar, though, dragonfly will fail to load it with the following error:

GrammarError: Two rules with the same name 'RuleA' not allowed.

How did this happen? We even renamed the extras! It turns out that each subrule instantiated in RuleRef is registered as a separate rule. By default, each instance will assign name = SubRule.__name__. Consequently, you’ll have to instantiate the subrules with unique names each time you re-use them. Fun fact: those names don’t have to bear any relation to anything else.

    extras = [
        RuleRef(rule = RuleB(name = "Sweeney Todd"), name = "rule_b1"),
        RuleRef(rule = RuleA(name = "Les Miserables"), name = "rule_a1"),
    ]

There are many like it, but this one is mine

I’m sure this is not the only way to do it: one could override the _process_recognition method of your MainRule, or perhaps caster, aenea, or dragonfluid implement equivalent nesting functionality in ways that I have overlooked. I would be very excited to learn about other approaches!

For now, I’m looking forward to applying this in my vim-grammar for dragonfly project. I’m hoping to write about the reasons why vim is excellent for voice programming later.

Executing nested rules with dragonfly

Introducing git to scientists who code

Many scientists treat coding — tasks, analysis, you name it — as a necessary evil we have to do in order to get to the science. You might know the result from your own scientific practice: subtle changes to the code strewn across many folders, days spent getting into the mind of the postdoc author five years gone, and a general unwillingness to touch the code unless it’s time to re-use it.

Before joining a research lab, I was lucky to have spent several years as a back-end programmer with the Yale Student Developers. (Best work-study job ever.) Consequently, working without proper version control and thorough documentation of each step now just feels icky.

(This isn’t my personal quirk, by the way. “Use version control” is the fifth recommendation of both Aruliah et al.’s Best Practices for Scientific Computing (2012) and Wilson et al.’s Good Enough Practices for Scientific Computing (2016). So, at the very least, I can say that I share a squick with a number of published researchers.)

Since joining Yale Decision Neuroscience Lab, I’ve been inducing colleagues to use git. The presentation that I gave yesterday is a very high-level overview of what git is good for:

(It doesn’t hold a candle to Alice Bartlett’s excellent Git for humans, which I heartily recommend, but it does use our lab’s problems as illustrations.)

Where do we from here?

Giving a tech presentation is only the first step in tech adoption. I lack a detailed roll-out plan, but here’s what I’m doing now:

  • Since jumping straight to the command line might be too scary, I’ve been recommending GitKraken and GitHub Desktop.
  • Any code I touch lands in a remote repository rather than the lab file-share. If someone wants to use it, that’s where they’ll get it. If I’m asked to with anyone’s code, it will need to be on that remote. This is on the theory that necessity is the best incentive.
  • And, of course, I’ve made it clear that anyone who struggles with anything git-related can contact me at any time and I’ll do my best to help.

We’ll see how it goes.

Introducing git to scientists who code

The Hidden Curriculum

This winter was longer than it had any right to be.

It started with the cold that froze my face, the snow that covered the campus, and the hope that every new semester brings. But the cold grew colder, and coldest still on the January day when Luchang Wang took her life.

I wasn’t close to Luchang, but I knew her: we worked together in a small group, aiming to make the world as good as it can get. I regret not being closer to her, because by all accounts, she was a brilliant, amazing friend. I got just a glimpse of it on a ride to Boston, when we ran through our friend Aaron’s Goodreads account. We talked Ayn Rand, and how Atlas Shrugged is my secret crack, and how rapey the Fountainhead is; the ride went by too quickly. The last time I saw her, she came late to our group meeting, but still she came, visibly tired and not entirely there. She jumped to her death from the Golden Gate Bridge four days later.

It wasn’t until her wake in Battell Chapel that I realized how many friends we had in common, how many of my frequented groups she belonged in, how much of who she was I aspired to. The reverberations of her passing went deep: the shockwaves it sent through our community took weeks to calm. They made this winter a winter of discontent.

*

Our anger didn’t replace sadness.

We raged at Yale for not making Luchang’s life just a little easier, in the hope that greater ease would ward off the void and the hopelessness and the pain. We saw incompetence and we were quick to label it systematic, because there was a system and it hurt some of our friends and one of our friends was gone, forever gone, and this was the only thing we could do. Maybe making it a little better for others could quench our rage and heal our hearts.

Maybe.

*

People talk about the hidden curriculum at Ivy League schools.

The theory goes that classes don’t matter as much as the skills you pick up from the elite environment, skills that the state-school riffraff can’t pick up because state schools are not the elite tower of pretentious assholes. They don’t wine and dine with the members of the elite, so they’ll never join their ranks.

That might well be true. But perhaps there is another set of lessons I would not have picked up at a Czech university. Namely: How to cope with the suicide of a friend you admired. How to handle the fear that your friends and loved ones might follow suit. How to survive battles with yourself that you only stand to lose.

How to face the void and stare it down.

It helps to realize that elsewhere is fine, too; that even if our lives will never be as good as they are now, they will still be good enough. It helps to realize that even if the top of Yale’s totem pole is hopelessly far, our goals do not require us to reach it. It helps to realize that we have friends, and we can make new ones, and we can fight battles together.

*

Luchang still lingers around the edges of my mind.

Just two days into Spring Break, I was passing through Berkeley’s deserted halls – and as I swiped into the computer cluster, I remembered when Yale Security told us Luchang’s last Yale ID swipe happened days ago, she was not on campus, our search was useless and her fate out of our hands. This will happen less and less, but I doubt it will stop completely.

This wasn’t helped by the fact that I chose to spend my spring break in San Francisco.

Call me stupid, but I honestly hadn’t realized until that moment that that was where the Golden Gate Bridge is. To attend a Harvard-Yale dodgeball trampoline game, I walked two miles with the bridge dominating the skyline as the sun was setting; I almost turned around and skipped the game.

It took me a week to muster the courage to actually go. It felt like I had to; if nothing else, as a form of exposure therapy. Plus, I’d be rewarded by an unprecedented view of the skyline! Except that on the day of, the entire bridge was enveloped in impenetrable fog. I should have known; it has its own Twitter account.

I walked the entire length of the bridge there and back. The fog itself never dissipated; it just made the next “THERE IS HOPE, MAKE THE CALL” crisis counseling sign on every other pylon pop out of nowhere.

In the end, I emerged out of the fog into the sunny San Francisco. I’m not sure if the trip was worth it.

*

It is the end of March, two months later now. Spring has come to campus, wavering, uncertain, frail. The snow melts into puddles and the puddles seep into the ground and soon, we’ll put the coats and scarves away. Spring has come.

But this winter still marches on.

The Hidden Curriculum