Challenges with learning end-user programming: A case study with Chemistry undergraduate students

This is an overview of the research paper Recommendations for Improving End-User Programming Education: A Case Study with Undergraduate Chemistry Students, published in the American Chemical Society (ACS) Journal of Chemical Education.

The work was conducted at Cal Poly and was led by MS student Will Fuchs. Our co-authors were Dr. Ashley McDonald from the Chemistry & Biochemistry department at Cal Poly and Dr. Aakash Gautam from the CS department at the University of Pittsburgh.

This article should be of interest to educators in disciplines other than computer science or software engineering who are teaching their students programming. For more details, see the paper.

Where should we teach Chemists to program?

Professional software engineers are a minority of professionals who use programming in their work. They are vastly outnumbered by professionals in other disciplines who use programming in their work (like chemists, climate scientists, or graphic designers). These professionals are often the end-users of the software they create, and they are referred to as end-user programmers. Students preparing to enter these roles tend to learn some programming at the undergraduate level, through various channels, each with its own challenges.

They could take the same intro programming course as computing majors, but this means they would be learning programming that is mostly disconnected from their discipline-specific context. An additional challenge, at least at Cal Poly, is that computing departments tend to be over-enrolled and under-staffed. If whole majors begin requiring their students to take our introductory CS course, we would quickly buckle under the enrolment pressure.

They could take specially-designed contextualised computing courses, where they learn programming specifically geared toward their discipline. For example, the Media Computation courses at Georgia Tech have been hugely successful at getting folks from different (demographic as well as professional) backgrounds interested in programming, and “CS + X” programs are sprouting up at universities all over the USA. This is great, but difficult for a campus to operationalise for all its majors that would use programming. If these courses were designed to cast a wide-enough net, we’d end up back at problem #1 above.

They could learn programming within their discipline-specific courses. An obvious challenge here is that most courses are already bursting at the seams with content, and the designers of those courses are often loath to let go of much of it to make room for new content. I’m the same—when I design a course I could swear that every single topic is absolutely essential. But the reality is that about 85% of the material is essential, and the rest is just, like, my opinion, man.

So, if we want to add programming to discipline-specific courses to prepare, say, chemists to use programming in their work, we need Chemistry professors who know how to program, and who care enough about its importance in their discipline to include it in their already-full curricula.

Happily, that describes the Physical Chemistry instructors at Cal Poly SLO.

Typically during their third year of undergraduate studies, students majoring in Chemistry or Biochemistry at Cal Poly take a 3-course sequence of Physical Chemistry courses, in which they also learn programming.¹ Including programming alongside thermodynamics, kinetics, and mechanics certainly does add to the instructional obligations of these courses, but over a decade with the curriculum has also revealed desirable gestalt effects—students not only learned to program; they also learned Chemistry through programming. Many students go on to successfully apply their newfound computing skills to real problems in their research labs at Cal Poly and beyond.

However, many students also seem to intensely dislike the programming aspects of the courses: they find it challenging, are unmoved by arguments that it would benefit them professionally, and generally don’t “reach for computing” to help them solve Chemistry problems outside the classroom. Most of them also self-select out of future computing experiences, opting out of an elective Computational Chemistry course following the 3-course Physical Chemistry sequence.

Understanding students’ challenges with learning programming

We (Chemists and Computer Scientists at Cal Poly) conducted two studies to learn about the challenges that the students faced while learning programming in these Chemistry courses. In the first study, we used surveys to learn about students’ attitudes toward programming and their programming abilities. In the second study, we conducted one-on-one interviews to gain deeper insight into their attitudes and abilities.

Preliminary surveys

We used the attitudes toward computing survey to measure students’ self-reported confidence with and enjoyment of computing, their perception of computing as being important and useful, and the strength of their sense that they belong in computing.

Results were…not fun. On average, the students reported negative attitudes toward computing: low confidence, low enjoyment, and low sense of belonging to a computing community. They did perceive computing to be marginally useful.

To measure students’ abilities with programming, we used the MCS1 concept inventory, a peer-reviewed assessment of MATLAB knowledge. (It’s a multiple-choice test.)

Students generally performed poorly on the assessment. Part of this is attributed to the fact that it included some language elements that they had not seen before. But there were also challenges on questions that “should” have been doable.

Interviews

Ok, so the students had negative attitudes toward computing, and struggled with MATLAB questions. That’s kind of all we can say from numerical responses to Likert questions. We can say nothing about what causes these negative attitudes, or what exactly their misunderstandings are that lead to their difficulties on the test.

Will, the MS student leading the work, conducted one-on-one interviews with 8 students enrolled in the Physical Chemistry courses. He asked them questions from the MCS1 assessment, and instead of framing them as multiple-choice questions, he framed them as open-ended short-answer questions, and asked the students to “think out loud” while answering them. He recorded the interviews (with the students’ consent), and together we analysed the transcripts.

Details are in the paper, but here are the highlights:

• We used Fuller et al.’s Matrix Taxonomy to categorise the level at which students were operating in terms of their programming knowledge. Students were generally competent with code comprehension involving patterns they had seen before.
• This served them well when they saw similar-looking programs, but stymied them when they were faced with different representations of the same concepts. They lacked an abstract enough mental model of MATLAB’s syntax and semantics.
• Without having been taught explicit strategies to solve programming problems, students faced challenges with program planning. This caused difficulties with multi-step problems, which were discouraging and daunting.

For example, consider the following MATLAB expression (a Boolean AND): and(a, b)

The students had previously only seen Boolean expressions in the context of conditional branching using if statements.

So, students who were correctly able to trace code involving conditional branching (e.g., if and(a, b)) were then unable to trace code that included c = and(true, and(true, false))

Lacking abstract understandings for statements and expressions, students combined the notions of if and and into a single construct based on superficial characteristics of programs they had seen before. Students faced similar difficulties with things like matrix concatenation and functions.

In addition to questions adapted from the MCS1, Will also asked the students about their feelings toward programming in general, and their confidence with the programming they had learned. He found that MATLAB appeared to be little more than a glorified graphing calculator to most of the students. The students’ collective mindset was exemplified nicely by this quote from one participant:

So I literally threw my calculator away and started doing everything on MATLAB.

Most students did not foresee themselves using MATLAB in their futures, and felt that they had not gained generalisable ability with programming that they could use outside the classroom. Some more quotes from students are below:

I know how to do explicitly what I’ve learned, and not much else.

I think with Chemistry they do a really good job of teaching us. But there’s no, like, okay we’re only going to be doing like MATLAB and like learning the very basics of the basics.

I should probably understand everything computationally little bit more because that’s the way that the research is moving and I think that’s, you know, an important part of the research future. So I want to develop these skills more but I’m like I don’t know what specific applications really look like.

Final remarks

Our paper closes with some recommendations for teaching programming to these would-be end-user-programmers:

• Promote abstraction and abstract understandings. We discuss strategies for helping students achieve abstraction, for example, using multiple representations of the same concepts to help disentangle students’ understandings from the specific examples they have learned with.
• Decomposition. Use sub-goals to break programming problems into small steps, and show examples of function reuse.
• Meta-cognitive awareness. Teach an explicit programming problem-solving process so that students may confidently approach larger multi-step problems. This should include steps for problem comprehension, testing, and debugging.

If you’re thinking that these recommendations would not be out of place in a book titled How to Teach Programming to Anyone, Not Just These Chemistry & Biochemistry Students, you’re right! These are all things that we try to accomplish in the introductory programming courses taken by our CS and SE majors. But these skills as they relate to programming tend not to be prioritised in end-user programming education. However, though these students are not aiming to become software engineers, they will engineer software, potentially of a critical nature, and their programming education should account for this.

The good news is that upper-division Chemistry and Biochemistry students already have significant skill with abstraction, decomposition, and meta-cognitive awareness. These skills are obviously not unique to computer science or programming—they are practised all the time in other STEM disciplines. And so our final recommendation is to harness the students’ and instructors’ existing competencies with abstraction, decomposition, and meta-cognitive awareness while teaching them in the context of programming.

For example,

• Abstraction is already used when reasoning about chemical equations, which are lightweight abstractions of real-world phenomena.
• Decomposition has been espoused as an important skill even in very early Chemistry education, e.g., scientific modelling skills involving breaking down a complex system into smaller elements and mechanisms.
• Meta-cognitive awareness is required to successfully navigate the multi-step process involved in identifying a chemical compound’s structure from its spectroscopic data.

The programming aspects of the Physical Chemistry courses are being revamped as I write this (including a transition to Python)—more updates soon!

---

Postscript

Why did we publish this paper at the Journal of Chemical Education, and not at a computing education venue like Koli Calling or SIGCSE?

I am confident that this paper is published at the right venue, but it took a bit to get here.

The manuscript was rejected from computing education venues twice! Both times, the primary (but not the only) hit against it was that it was probably of more interest to Chemistry instructors than CS instructors (the implication being that the readership of computing education research publications are mostly computing instructors). I have two reasons to push back against this. First, my view is that our Chemistry faculty who are teaching programming are computing instructors — they are teaching computing! Second, I perhaps naïvely expect that if instructors in other disciplines are interested in research about teaching computing, they would look in computing education research publications for recommendations or techniques.

(Ok, fine, I also have some sense of professional identity wrapped up in publishing at my “home” venues.)

Meanwhile, as part of her broader work in computational Chemistry education, our co-author Ashley had been working with her colleagues to update the curriculum based on this work. She’d been travelling to meetups of Chemistry educators and talking about this work, and reported high levels of interest and engagement from other Chemists working to include computing in their courses. The audience for the paper was right there.

So the question was: do we choose a home for the paper based on its topic area, or its target audience? In most cases, these point to the same venues. But that’s not always true for works that straddle multiple academic disciplines. Choosing a venue for a target audience may require less “convincing” to get it published, but choosing for a topic area may mean people can find it easily in the future. I don’t know that there’s an obvious answer, but in this case there was certainly a practical one.

We made significant improvements to the manuscript (for example, we added the entire discussion section on abstraction, decomposition, and meta-cognitive awareness) and submitted it to the Journal of Chemical Education, where it ended up with extremely positive reviews.

I’m also grateful to be in a department where my scholarly output is not expected to exist in a narrow selection of publication targets—without this, there would’ve been a whole other calculus to consider.

---