There has been lots of interest in gpt-2 recently - it's taught itself to "talk" and to play chess.  If you've not heard about it, it's a neural network that has learned to produce language using a massive database of language samples scraped from the internet and a huge number of tune-able parameters, but without any explicit knowledge about the rules of language. 

As Gary Marcus points out in this excellent and accessible article, gpt-2 offers a new testing ground for theories of language acquisition and whether humans come pre-wired with tools for learning language or whether language can be learned from the input statistics alone. 

I've just begun messing around with the model using the Python package: gpt2-simple.  It's nice and easy to use, the only snag was having to depreciate TensorFlow to an older version to make it run.  I used the "small model" - trained on 8 million web pages - referred to as 124M and fine-tuned the model "for fun" with the conference program of the Society for Neurobiology of Language Conference 2019.  I ran 1000 training iterations without using GPU and it finished in around 5 hours on an Ubuntu system: Intel® Core™ i5-8400 CPU @ 2.80GHz × 6 (8 GB RAM). 

So here is what gpt-2 thinks that language science is all about ...

The all important welcome information, after all, every conference program needs a strong opener:

"Welcome to the Online SNL 2019 Program! SNL 2019 is upon us! SNL 2019 brings together the leading researchers from different disciplines in a game-changer for neurolinguistics in neurotypicalism. This includes a whole new venue for neurolinguistics in neurotypicalism, a whole new venue for neurolinguistics in neurotypical behavior, and a whole new venue for neurolinguistics in the brain.  Participants from around the globe travel to South America, the Caribbean Sea, and the Indian Ocean to attend the Opening Night Reception, which features some of the best and the worst from across the neurolinguistics fields."

On a more serious note, gpt-2 does a decent job with poster titles: 

D84 Lingering Pronunciation Constraints during Naturalistic Intervals of Word Form

B76 Hemispheric parallel access for orthographic comprehension

B77 Interactions of orthographic depth and semantic complexity in proficient spoken and written language Comprehension.

B3 Prediction: Agents and Predictors of Reading

B33 Sparse canonical correlation analysis of reading: when and how?

A38 Steady state analysis of the fMRI fMRI ERP lexicon

E22 A new perspective on pathology in post-stroke aphasia: Investigating the processes underlying recovery from stroke and improving surgical outcome is an essential first step in translational therapy trials

B53 Rule learning in reading: Evidence from ERPs and PET

E85 Distinct neural substrates of visual and auditory processing converged in temporoparietal
cortex during L2 activation

And also pretty good on full abstracts (although note, as Gary Marcus' article points out not every thing that comes out is sensible, I'm cherry picking the better ones):

"A key question in the neuroscience of language is whether masses of individuals engineer their own thought processes via algorithmic processes or they form independent trials driven by their own algorithmic decisions. Theories of Brain and Mind are diverse; some posit that a core function of language production is in the making of conceptual sentences; while other views propose that language production is a collection of experiential functions, that it must choose its objects, events or minds and therefore must choose between semantic and introspective features. The present study is to investigate whether a core function of language production is composing conceptual sentences in a non-attentional capacity. We used a fast-event inference (FIF-ICA) paradigm to investigate the neural bases of this challenging behaviour. We used a fast-parse approach to investigate the neural bases of this challenging behaviour. We studied 150 healthy young volunteers for about five minutes whilst viewing pictures of familiar objects. The participants had difficulty forming coherent sentences as a result of aneuplasia of the premotor cortex (PMC). We observed two patterns of activation in our analyses of these images a few hours after the face-off. A central region of interest (COWI) was positively disrupting when object naming produced differences between the two groups of healthy volunteers. Diffusion-tensor imaging (DTI) data were acquired for all participants at a resolution of 1.5mm by 2mm. We focused on regions of interest (ENC) which direct visual and auditory cortices, implicated in language production. We observed a relatively widespread activation pattern across the cortical hemispheres, with disproportionately distributed areas of the bilateral hemisphere in healthy individuals. While there was limited evidence in the previous a few years that these areas are sensitive to cognitive changes, we now showed that these encodings are gradually becoming more detectively correlated over time. We found that the bilateral lateral cerebellum, which is involved in visual word production, was also being observed to be affected by these changes. We discussed below the neuroimaging results with the aim of identifying regions of interest in the context of picture-naming that might be related to these neuro-treasuring processes. The regions of interest were identified using fBOI, which is a resting-state functional-functional magnetic resonance imaging (rfMR) computerized imaging tool. More specifically, we investigated the effect of changes in resting-state functional connectivity (rsFCC) on picture naming in healthy ageing. We found that over the presence of aged objects, rsFCC became more dependent on predictive coding than on any other domain in the domain. We also found that rsFCC evoked a greater response of the WABC than any other domain in the >4-year-old group. Thus, a core function of language production is being spatially and spectrally correlated with the development of language production. Future studies on this subject will be emphasized at both the state and domain level."

"The neurobiology of the phonological ratio, between Heschl’s and Deaf listeners, is a debated topic in neurobiology. In the current study, we used magnetoencephalography (MEG) to understand the timing and correlates of the acquisition of phonological sequences. Twenty-four participants (age 16-49, range 23-64) were tested in a visual novel Word-to-Word trial with three conditions: (1) an articulatory target word with a lexical layout phonologically associated with the letter / / (versus /sh//), (2) a non-agricultural target word with a lexical layout phonologically related to the letter /i/ (versus /s/), and (3) a semantic target word with a phonological revision based on the letter /i/ (versus /sh/). The patterns of detection were further used to be refined to include the components of phonological integration, termed “phonological artifacts”. These components, known as “non-agricultural artifacts”, were acquired as free samples, and were quantified using MEG. Non-agricultural artifacts were identical in both hemispheres. Non-European languages (e.g., Finnish) were used as controls. Both parental language (English) and her native language (Norwegian) were used as controls. The brain activity of the non-agricultural artifacts was analyzed by calculating the inter-hemispheric disconnection (IHDC) between the cortical areas supporting the phonological processing of the pitch and frequency domain. We found that the cortical circuits supporting phonological word retrieval were disconnection induced by the presence of the non-agricultural artifacts in the same regions as the cortical areas supporting word identification, namely, the basal, middle, and inferior temporal language areas. In addition, we observed that the cortical circuits supporting syntactic processing, namely, the dorsolateral and ventral language areas, were disconnection induced by the presence of the non-agricultural artifacts in the same regions as the cortical circuits supporting phonological processing, namely, the insula and basal language areas. Direct cortical mapping of the inferior and middle temporal areas is proposed to support the development of the reading network."

So that's decided, next year gpt-2 is writing my abstracts whilst I put my feet up...I'm also happy to lend my trained model to the SNL organising committee, for a small fee.

What happened this year?  Lots of good things.  Two great publications.  One in the new open access journal recently launched by the Society for the Neurobiology of Language (The Neurobiology of Language).  This was work led by Evelyne Mercure and you can read it here.  Publishing in this new journal was a great experience.  The journal has a consensus review model such that the handling editor synthesizes the reviews from reviewers and revisions are agreed among the panel before passing to the authors.  This seems to work well and prevents conflicting suggestions of further work from reviewers with different perspectives.   What was the study about?  Using fNIRS, we measured neural responses of monolingual and bilingual infants (both sign-speech and spoken language bilinguals) to familiar and unfamiliar spoken and sign languages to understand how neural systems for language differ  in bilinguals, of particular interest were neural systems in hearing children born to deaf parents, that are bilingual in sign language and speech, that likely don't have the same exposure to parental speech in-utero.   One important finding was that unimodal bilingual experience was shown to have a greater impact on early brain lateralization than bimodal bilingual experience.

The other study was published in Current Biology and can be found here.  In this study using fMRI we looked at whether the same semantic representations were accessed in adult sign-speech bilinguals when they attended to the same conceptual items presented as sign and speech.   Interestingly we found that category representations, e.g. whether something was an animal or a fruit, etc., were similar but not neural patterns for individual concepts, e.g. the pattern for individual spoken words and signs were not identical.  This suggests that the precise language that you use likely influences the way that you understand the world. Karen Emmorey wrote an excellent commentary on the article which can be read here.

We also published a pre-print of a large scale speech perception experiment from my time working Cambridge.  Hopefully this can be shepherded to a journal near you soon... You can read it here.  This was a massive study in which we tested over 1500 participants on a speech categorization task using a wider variety of phonemic contrasts than have been previously tested.  The study shows that speech perception skills continue to develop until late childhood/early adulthood (~17 years of age) and demonstrates the role that lexical information has in shaping perception. 

Other things that happened, Kathryn Hewson joined the lab as a PhD student working on studies of swearing and taboo language processing, we are looking forward to presenting this new line of work from the lab soon.  We finalized the design of a multi site experiment for the language consortium that I helped to establish.  We begun pre-registering BSc student projects and started our ReproducibiliTea journal club.  Looking forward to what 2020 brings...

We recently presented our work at the Neurobiology of Language Conference in Helsinki.  It was a great meeting.  It's helped to generate lots of ideas for new studies.  Watch this space!

Excited to announce that the CALC lab is gaining a new member.  Kathryn Hewson will be starting a PhD with us in September, working on the acoustic, perceptual and physiological effects of swearing.  Watch this space for exciting work in this area.  If you would like to come and work in the lab send me an email.  Like in Kathryn's case, we can work up a competitive PhD studentship proposal (and also vacation scholarships) together to get internal or external funding. 

I returned to DCAL last week to give a talk on our new preprint: https://www.biorxiv.org/content/10.1101/623645v1.  It was great to get some feedback on the study - lots of great ideas about what to do next and interesting new perspectives on our data.

At the moment there are fully funded PhD positions currently available in our department.  These are available via open competition.  If you would like to submit a PhD application to work with me, then please contact me via email: S.Evans1@westminster.ac.uk.  See here for further information.

I will be giving a talk at the BPS "teaching reproducible research"  workshop on the 28th November.  The workshop provides University lecturers and pre-tertiary teachers with an introduction to the reproducibility crisis facing Psychology. It outlines how psychological science is responding to this challenge, and presents case studies of how psychology education is adapting.  If you want to book a place click here: https://www.kc-jones.co.uk/dartp2018​
​

Our recent work, led by Saloni Krishnan, looking at the brain responses of guitarists and beatboxers when they listen to music has been published in Cerebral Cortex.  Here is a link to some of the recent press about the study and the study here.

I'm giving an invited research talk at the MAPS summer school in Groningen.  There is a great line up of speakers including Anne Cutler, Elia Formisano, Sonia Kotz and others.  Click here to find out more: https://www.rug.nl/education/summer-winter-schools/summer_schools_2018/maps/